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FOREWORD 

Use  of  a  system  is  always  accompanied  by  a  risk  of  failure,  which  may  arise  from  deficien¬ 
cies  of  design,  manufacture,  materials  or  other  essentials.  The  underlying  purpose  of  any  test 
is  to  evaluate  that  risk  in  whole  or  in  part,  to  assess  the  magnitude  and  acceptability  of  the 
risk  by  qualitative  or  quantitative  means. 

Before  and  after  a  Strategic  Weapons  System  is  deployed,  a  great  many  elements  of  risk 
must  be  evaluated  by  testing.  In  total,  tests  may  occupy  an  appreciable  part  of  the  facilities 
and  manpower  committed  to  a  procurement  program  and  may  account  for  a  substantial  part 
of  the  program’s  cost. 

The  objective  of  this  manual  is  to  provide  Strategic  Systems  Project  Office  (SSPO)  con¬ 
tractors  with  program  guidance  for  structuring  and  administering  test  programs  in  accordance 
with  the  applicable  requirements  of  NAVSEA  OD21549A,  Technical  Program  Management 
Requirements  for  Navy  Strategic  Systems  Project  Office  Acquisitions.  Where  inputs  are  re¬ 
quirements  of  NAVSEA  OD21549A,  they  are  so  indicated. 

The  techniques  of  this  manual  have  been  applied  successfully  for  many  years.  In  particu¬ 
lar,  the  use  of  a  permanent  Integrated  Test  Program  Board  as  a  test  program  management 
device  has  been  demonstrated  by  experience  in  many  programs,  experience  showing  that  the 
collective  evaluations,  judgments  and  decisions  of  an  Integrated  Test  Program  Board  are  likely 
to  be  better  informed  and  less  subject  to  bias  than  those  made  by  individuals.  On  the  other 
hand,  when  specific  actions  are  required,  it  has  been  found  more  efficient  to  assign  them  to 
individuals  rather  than  to  accomplish  them  through  the  agency  of  a  board.  Operating  policies 
recommended  herein  embody  this  basic  policy  for  using  the  knowledge  and  experience  of  a 
group. 

Procedures  given  herein  are  compatible  with  the  analytical  approach  of  NAVSEA 
OD29304B,  Reliability  And  Availability  Evaluation  Program  Manual.  With  few  exceptions 
the  terminology  conventions  of  that  document  have  also  been  observed. 

This  revision  of  NAVSEA  OD42282  has  been  prepared  to  provide  additional  material 
relating  to  management  of  test  programs,  and  to  extend  the  coverage  of  the  1973  edition  to 
encompass  testing  of  computer  software  and  technical  documentation,  as  well  as  planning  for 
testing  in  post-development  program  phases. 
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BASELINE 


COMPONENT 


DEMONSTRATION 

EQUIPMENT 
ERROR  RATE 

ESTIMATION 

FAILURE 

FAILURE  RATE 

FIRMWARE 

ITEM 

INTEGRATED  TEST  PROGRAM 


i' 


MAINTAINABILITY 


GLOSSARY 

A  description  of  requirements  or  configuration 
defined  by  specification,  to  which  subsequent 
changes  are  related. 

A  combination  of  parts,  devices  and  structure, 
usually  self-contained,  which  performs  a  distinct 
function  (acts  on  one  or  more  inputs  to  produce  an 
appropriate  output)  in  the  operation  of  an  equip¬ 
ment;  for  example,  a  converter,  gas  generator, 
amplifier. 

Formal  measurement  of  system  characteristics  with 
statistical  confidence  by  testing  or  operation.  Both 
estimation  and  hypothesis  testing  approaches  are 
used. 

The  first  assembly  level  below  a  subsystem;  for 
example,  a  Digital  Geoballistic  Computer. 

The  average  rate  at  which  software  errors  appear 
in  real-time  operation  under  mission  conditions. 
Analogous  to  hardware  failure  rate. 

The  use  of  testing  to  form  estimates  of  population 
parameters  and  to  evaluate  the  precision  of  those 
estimates. 

Performance  below  a  specified  minimum  level  or 
outside  a  specified  tolerance  interval. 

For  devices  described  by  the  exponential  model, 
the  positive  constant  X.  It  is  the  reciprocal  of 
mean-time-between  failures. 

Programs  residing  in  PROMs,  ROMs,  etc. 

General  term  denoting  physical  element  of  a 
system  (any  assembly  level). 

A  test  program  in  which  each  test  contributes  to 
the  timely,  cost-effective  achievement  and  verifi¬ 
cation  of  product  requirements. 

A  measure  of  the  ability  of  an  item  to  be  main¬ 
tained.  Mean  preventive  maintenance  time  and 
mean  repair  time  are  commonly-used  indices  of 
maintainability.  Maintainability  is  sometimes  de¬ 
fined  as  the  probability  of  repair  within  a  stated 
time. 
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MILESTONE 

A  planned  definitive  event  during  ,i  program  lor 
example,  the  completion  of  a  major  work  elemeni 

MODULE,  HARDWARE 

An  onboard-replaceable  item;  for  example,  a  I  >  p  • 
3  Module. 

MODULE,  SOFTWARE 

A  unit  of  the  hierarchical  organization  ot  a  soft¬ 
ware  program,  in  which  a  major  program  function 
is  accomplished.  A  module  has  identifiable  bound¬ 
ary  statements  and  can  be  referenced  by  name 
from  other  parts  of  the  program. 

PRIME  HARDWARE 

Hardware  manufactured,  inspected,  tested  and 
handled  in  full  compliance  with  all  specification 
requirements  and  all  material  and  process  controls 
applicable  to  operational  hardware. 

RELIABILITY 

The  probability  that  an  item  will  perform  its  in¬ 
tended  function  without  failure  for  a  specified 
interval  under  stated  conditions,  given  that  it  is 
up  (operable)  at  the  beginning  of  the  interval. 

RISK,  CONSUMER’S 

The  probability  that  a  test  will  accept  by  chance  a 
device  or  lot  having  a  characteristic  equal  to  a 
specified  unacceptable  level.  A  variety  of  Type-II 
error. 

RISK,  PRODUCER’S 

The  probability  that  a  test  will  reject  by  chance  a 
device  or  lot  having  a  characteristic  equal  to  a 
specified  desired  level.  A  variety  of  Type-I  error. 

SCREENING 

Tests  or  inspections  applied  to  1 00%  of  product  to 
precipitate  latent  defects  or  to  improve  the  average 
reliability  of  outgoing  products. 

SOFTWARE 

Computer  programs  and  data. 

SUBCONTRACTOR 

A  supplier  or  vendor  to  a  contractor. 

SUBSYSTEM 

The  first  indenture  level  below  a  system.  For  exam¬ 
ple,  the  Navigation  Subsystem  or  Fire  Control  Sub¬ 
system  of  the  Strategic  Weapon  System. 

SYSTEM 

A  collection  of  functionally  related  items,  which 

together  perform  one  more  useful  functions;  for 
example,  the  Strategic  Weapon  System. 


A  procedure  for  evaluating  one  or  more  of  the 
attributes  or  variables  of  an  item.  A  test  frequently 
involves  operating  the  device,  and  usually  involves 
deliberate  application  of  stress,  either  externally 


TEST 
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TEST  (Continued)  (e.g.,  environmental  stress)  or  internally  (e.g.,  self- 

induced  stress).  The  term  test  may  also  include 
passive  examinations  or  inspections  of  an  item  as 
applicable. 

TESTS,  ACCEPTANCE  Tests  performed  to  determine  the  acceptability  of 

outgoing  product.  Normally  applied  to  100%  of 
product,  but  sometimes  performed  on  lot  sampling 
basis  when  tests  are  destructive. 

TESTS,  AGING  Tests  performed  to  detect  and  track  ogressive 

irreversible  changes  brought  about  by  a. 

TESTS,  DEVELOPMENT  Tests  performed  on  preliminary  or  proi  ;•<;  hard¬ 

ware  to  determine  design  and  performa  parame¬ 
ters. 

TESTS,  ENGINEERING  EVALUATION  Functional  environmental  tests  pen^med  to 

evaluate  characteristics  of  product  design  and  to 
determine  compliance  with  performance  and  en¬ 
vironmental  requirements. 

TESTS,  QUALIFICATION  Tests  performed  to  demonstrate  that  prime  hard¬ 

ware  meets  design  specification  requirements  in¬ 
cluding  mission  environments. 

TESTS,  QUALITY  ASSURANCE  Tests  conducted  to  verify  conformance  to  quality 

requirements  and  to  determine  acceptability  of 
product. 

Tests  performed  to  retain  or  regain  qualified  status 
of  a  product  after  any  of  the  following:  a)  change 
in  hardware  design,  b)  change  in  source,  c)  change 
in  manufacturing  processes  or  plant  location, 
d)  production  interruption  for  a  length  of  time  such 
that  continued  validity  of  previous  qualification 
becomes  suspect,  e)  disqualification  of  a  product, 
f)  major  changes  in  test  equipment  or  test  proce¬ 
dures,  g)  major  changes  in  tooling,  dies  or  fixtures. 

Tests  performed  to  assess  software  compliance 
with  design  requirements.  Validation  tests  include 
module  tests,  module  integration  tests  and  system 
level  tests. 

TESTS,  VULNERABILITY  Tests  designed  to  demonstrate  performance  capa¬ 

bility  after  exposure  to  radiation  environments. 


TESTS,  REQUALIFICATION 


TESTS,  VALIDATION 


VARIABLE 


A  characteristic  or  property  that  is  appraised  in 
terms  of  scalar  values. 
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CDRL  Contract  Data  Requirements  List 

CPCI  Computer  Program  Configuration  Item 

DASO  Demonstration  And  Shakedown  Operation 

EET  Engineering  Evaluation  Tests 
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MTBF 


MTTR 


PROM 


RM&Q 


Joint  Chiefs  of  Staff 


Mean  Time  Between  Failures 


Mean  Time  To  Repair 
Operational  Test 


Production  Assessment  Tests 


Programmable  Read-Only  Memory 
Qualification  Program  Plan 
Radio  Frequency 

Reliability,  Maintainability  And  Quality 
Root  Mean  Square 
R  ^ad-Only  Memory 
Single  Integrated  Operations  Plan 
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SITP 

SOTP 

SPALT 

SSPO 

SWS 

TVA 


Shipyard  Installation  Test  Program 
Shipyard  Overhaul  Test  Program 
Strategic  System  Projects  Alterations 
Strategic  Systems  Project  Office 
Strategic  Weapon  System 
Temporary  Variance  Authorization 
Weapon  System  Readiness  Test 
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Section  1 

INTRODUCTION 


When  the  requirements  for  an  overall 
Strategic  Weapon  System  (SWS)  and  for  the 
constituent  subsystems  and  equipments  have 
been  established,  it  is  essential  that  every 
effort  be  directed  at  meeting  those  require¬ 
ments.  NAVSEA  OD2 1549 A,  Technical  Pro¬ 
gram  Management  Requirements  For  Navy 
Strategic  Systems  Project  Office  Acquisitions, 
requires  contractors  to  establish  and  maintain 
an  Integrated  Test  Program  (ITP),  to  assure 
that  system  design  meets  requirements  (i.e., 
adequate  design  capabilities  and  design 
margins). 

An  ITP  should  feature  planning  and  man¬ 
agement  of  test  activities  during  system 
acquisition  and  later  program  phases,  so  that 
required  system  performance  confidence  is 
obtained  within  schedule  and  at  minimum 
cost  in  program  resources.  Careful  planning 
and  day-to-day  management  is  needed,  if  test¬ 
ing  is  to  be  coordinated  smoothly  with  other 
program  activities  to  permit  critical  program 
milestones  to  be  met. 

This  is  particularly  true  because  testing 
cannot  be  scheduled  rigidly.  Flexibility  is 
necessary  in  the  management  of  a  test  pro¬ 
gram  became  the  progress  of  testing  depends 
on  design  and  development  delays,  occurrence 
of  failures,  facility  limitations,  and  other  con¬ 
straints. 

An  important  element  of  an  integrated  test 
program  is  coordination  with  an  Integrated 
Data  System  (IDS).  Data  reporting,  analysis 
requirements  and  formats  should  be  reflected 
in  the  Integrated  Test  Program  Plan  (ITPP), 
and  in  individual  test  plans,  procedures  and 
reports. 

The  tests  required  to  assure  performance  of 
a  SWS  are  complex  and  extensive.  Manage¬ 
ment  concepts  are  described  in  this  manual 
for  use  during  the  development  and  produc¬ 


tion,  and  for  planning  the  pre-deployment, 
deployment  and  special  operations  phases 
of  a  program.  Three  facets  of  the  overall 
test  management  task  are  discussed  in  this 
manual- 1)  management  and  control,  2)  test 
planning,  and  3)  test  integration. 

The  contractor  is  responsible  for  assuring 
that  all  contract  requirements  are  met  or 
exceeded.  If  any  requirements  are  not  met, 
the  contractor  is  responsible  for  obtaining 
contract  relief.  Pursuant  to  this  objective,  the 
program  manager  is  responsible  for  the  overall 
management  of  the  test  program.  His  respon¬ 
sibilities  include  test  program  direction, 
changes,  policy  and  budget  authorizations, 
and  establishing  an  ITP  approach  to  meet  the 
requirements  of  NAVSEA  OD2 1 549A.  This 
document  utilizes  the  Integrated  Test  Pro¬ 
gram  Board  (ITPB)  concept  to  handle  the 
day-to-day  administration  of  the  test  pro¬ 
gram,  but  the  contractor  may  use  a  different 
organizational  technique,  if  desired,  to  accom¬ 
plish  the  planning  and  control  functions  this 
document  assigns  to  the  ITPB. 

The  ITPB  is  established  to  provide  central¬ 
ized  planning  and  administration  of  test  pro¬ 
grams  and  to  promote  optimum  use  of  test 
resources.  It  provides  a  forum  for  assembling 
and  evaluating  test  program  objectives,  plans 
and  problems,  for  resolving  conflicts  and 
inconsistencies  in  schedules  and  priorities,  for 
evaluating  and  reporting  progress,  and  for 
realigning  the  tests  as  program  changes  occur. 
The  ITPB  is  responsible  for  assuring  that  the 
tests,  as  performed,  enable  continuing  evalua¬ 
tion  of  performance  and  quality,  as  well  as 
supporting  corrective  action  decisions  and 
other  decisions  as  necessary. 

Responsibility  for  planning  specific  tests, 
performing  them  and  analyzing  the  resulting 
data  is  assigned  to  various  line  organizations- 
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typically  design,  test,  and  product  assurance 
engineering.  These  organizations  submit  plan¬ 
ning  information  (Figure  4-1)  to  the  ITPB, 
where  it  is  reviewed,  coordinated  and  ap¬ 
proved.  Approved  plans  are  summarized  in  an 
ITPP. 

The  test  planning  function  encompasses 
test  integration  across  the  range  of  program 
phases  in  which  the  contractor  has  test 
responsibilities.  Integrated  test  planning 
consists  of  two  basic  tasks,  selection  of  tests 
for  which  the  need  is  greatest  and  assurance 
that  the  tests  are  performed  in  a  way  that 
maximizes  the  information  obtained  in  return 
for  the  resources  invested,  so  that  contractual 
requirements  are  achieved. 

Planning  is  performed  iteratively  and  is 
refined  periodically  to  accommodate  neces¬ 
sary  changes  after  the  program  is  set  in 
motion.  Testing  progress  is  evaluated  contin¬ 
uously  under  the  administrative  direction  of 


the  ITPB,  which  issues  periodic  status  reports 
to  the  program  manager. 

This  document  covers  development,  pro¬ 
duction,  pre-deployment,  deployment  and 
special  operations  test  planning.  A  discussion 
of  software  testing  in  the  development  phase 
is  provided  to  emphasize  the  emergence  of 
software  as  a  major  element  of  systems  and  as 
a  major  source  of  test  program  problems,  such 
as  schedule  slippages  and  test  cost  overruns. 
This  document  also  places  special  emphasis 
on  reliability  and  maintainability  evaluation 
and  testing.  Although  reliability  and  main¬ 
tainability  are  performance  characteristics, 
they  are  afforded  special  consideration  herein, 
because  of  the  emphasis  placed  on  them  in 
NAVSEA  OD21549A. 

In  this  document  the  term  item  is  used  to 
represent  any  SWS  hardware  or  software 
element  that  has  performance,  design  and 
testing  requirements,  regardless  of  assembly 
level. 
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Section  2 

TEST  PROGRAM  MANAGEMENT  CONCEPTS 


2.1  PURPOSE  AND  SCOPE 

NAVSEA  OD21549A  requires  that  the 
contractor  establish  and  maintain  an  inte¬ 
grated  test  program  at  the  outset  of  the  de¬ 
velopment  phase.  In  the  development  phase 
the  program  should  cover  development  tests, 
engineering  evaluation  tests,  qualification 
tests,  software  tests,  reliability  and  maintain¬ 
ability  demonstrations,  as  well  as  planning 
for  production  tests  and  inspections,  pre¬ 
deployment  and  special  operations  phase  tests 
to  the  extent  of  the  contractor’s  responsibili¬ 
ties  in  those  later  program  phases.  Figure  2-1 
lists  test  documents  required  by  NAVSEA 
OD21549A,  with  the  sections  of  this  docu¬ 
ment  in  which  they  are  discussed. 

This  document  describes  the  integrated 
test  program  concept  and  practices  which  can 
be  used  in  the  development  phase  to  meet 
the  applicable  requirements  of  NAVSEA 
OD21549A.  It  also  addresses  testing  in  post¬ 
development  program  phases.  Planning,  moni¬ 
toring  administration  and  reporting  functions 
necessary  to  implement  an  integrated  test 
program  are  presented  as  guidance  to  aid  con¬ 
tractors  in  planning  and  executing  technically 
valid  cost-effective  test  programs.  This  docu¬ 
ment  does  not  establish  contractual  require¬ 
ments.  The  methods  discussed  herein  are 
structured  to  allow  continuing  visibility  of 
testing  activities  and  program  status  to  con¬ 
tractor  management  and  the  Strategic  Systems 
Project  Office  (SSPO).  Contractors  should 
tailor  their  integrated  test  programs  and 
supporting  activities  as  appropriate  to  the 
needs  of  their  particular  programs. 

One  of  the  first  contractor  activities  after 
a  contract  is  finalized  is  development  of  an 
Integrated  Test  Program  Plan.  Test  program 
planners  should  be  careful  not  to  change  the 


scope  of  contractually  specified  testing,  but 
to  provide  specific  program  definition  and 
details  of  test  design,  objectives  and  pro¬ 
cedures  consistent  with  contract  require¬ 
ments.  The  term  test  program  as  used  herein 
means  all  such  tests  considered  collectively. 

Preliminary  planning  for  testing  during  the 
later  program  phases  of  production,  pre¬ 
deployment,  deployment  and  special  opera¬ 
tions  (e.g.,  aging  and  surveillance  tests,  DASO) 
for  which  the  contractor  may  be  responsible, 
should  be  accomplished  during  the  develop¬ 
ment  phase,  and  the  ITPP  should  be  revised 
to  include  any  tests  planned  for  execution  by 
the  contractor  during  those  later  phases. 

Figure  2-2  lists  typical  contents  of  an  ITPP. 

2.2  INTEGRATED  TEST  PROGRAM 
CONCEPT 

In  an  integrated  test  program  all  of  the 
tests  contribute  without  void  or  unnecessary 
overlap  to  the  achievement  and  verification 
of  item  requirements  within  general  program 
goals  of  schedule  and  minimum  cost.  A  posi¬ 
tive  management  system  is  needed  to  elim¬ 
inate  duplicate  tests,  make  maximum  use  of 
test  information  and  assure  that  relevant  re¬ 
sources  are  used  effectively. 

The  integrated  program  concept  embodies: 

•  An  overall  test  program  philosophy. 

•  Clear  assignment  of  responsibilities  to 
participating  groups. 

•  Identification  of  all  design  requirements, 
and  the  sources  of  those  requirements,  that 
require  testing  to  assure  compliance  with  con¬ 
tractual  requirements. 

•  Verification  of  each  such  requirement  by 
specific  tests. 

•  Coordinated  test  program  scheduling. 

•  Decisions  on  the  methods  and  rationale 
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Documents  Defined  By  NAVSEA  OD21549A  Discussed  Herein  In  Section 

Integrated  Test  Program  Plan  2.2 

Integrated  Test  Program  Status  Reports  3.5.5 

Qualification  Program  Plan  3.6. 1 .3 


Qualification  Test  Plans 
Qualification  Test  Procedures 
Qualification  Test  Reports 
Software  Acceptance  Test  Plans 
Software  Acceptance  Test  Procedures 
Software  Acceptance  Test  Reports 
Software  Validation  Test  Plans 
Software  Validation  Test  Procedures 
Software  Validation  Test  Reports 
Engineering  Evaluation  Test  Plans 
Engineering  Evaluation  Test  Procedures 
Engineering  Evaluation  Test  Reports 
Reliability  Demonstration  Test  Plan 
Reliability  Demonstration  Test  Procedure 
Reliability  Demonstration  Report 
Maintainability  Demonstration  Test  Plan 
Maintainability  Demonstration  Test  Procedure 
Maintainability  Demonstration  Report 
Development  Test  Procedures* 

Development  Test  Reports* 


3.6. 1.3 

3.6.1. 3,  4.2.2. 1 
3.6. 1.3,  4.2.2.2 

3.6. 1.3,  4.2 .2.3 
3.6. 1.5,  4.3 
3.6.1. 5,  4.3 

3.6.1.5,  4.3 

3.6.1. 4,  4.3 
3.6.1. 4,  4.3 
3.6.1. 4,  4.2.2.3 
3.6. 1.2,  4.2.2. 1 
3.6.1. 2,  4.2.2  2 
3.6.1. 2,  4.2.2  3 

3.6. 1.6,  4.4 
3.6. 1.6,  4.4 
3.6. 1.6,  4.4 
3.6. 1.6,  4.5 
3.6. 1.6,  4.5 
3.6. 1.6,  4.5 
3.6.1. 1,  4.2.2.2 
3.6.1. 1,  4.2.2.2 


*For  tests  intended  to  evaluate  or  verify  design  capability. 


Figure  2-1.  Documents  Of  An  Integrated  Test  Program 
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TYPICAL  CONTENTS  OF  AN  INTEGRATED 
TEST  PROGRAM  PLAN 

*1.  Description  of  the  organization  and  management 
of  the  Integrated  Test  Program. 

*2.  Summary  of  planned  tests,  including  type  of  test 
and  test  objectives. 

3.  Samples  of  standardized  data  collection  forms  to 
be  used. 

*4.  Identification  of  tests  from  which  reliability  and 
maintainability  assessment  data  will  be  obtained. 

*5.  Schedules  for  tests,  relating  test  program  mile¬ 
stones  to  major  program  milestones. 

*6.  Schedules  for  special  test  facilities/equipment,  test 
items  and  test  documentation. 

7.  Government  Furnished  Material  requirements  and 
impact. 

8.  Identification  of  any  approved  deviations  from 
contractual  requirements. 

9.  Identification  of  applicable  operating  policies  and 
procedures. 

‘Required  by  NAVSEA  OD21 549A 

Figure  2-2.  Typical  Content  Of  An  Integrated 
Test  Program  Plan 


that  will  be  used  to  analyze  test  data,  before 
beginning  the  test. 

•  Standardized  data  collection  forms  and 
methods. 

•  Regular  reporting  of  test  status  and 
results. 

The  benefits  of  an  integrated  test  program 
include: 

•  A  common  point  of  high  visibility  from 
which  the  test  program  can  be  managed. 

•  A  test  requirements  baseline  for  each 
functional  test  used  to  verify  or  demonstrate 
the  design. 

•  Complete  uniform  collection,  analysis 
and  reporting  of  test  data  from  contractor 
and  sub-tier  supplier  test  sites. 

•  A  master  checklist  by  which  the  program 
manager  can  monitor  and  evaluate  contractual 
performance. 

•  Establishment  of  production,  pre¬ 
deployment,  deployment  and  special  opera¬ 
tions  phase  test  requirements  at  the  earliest 
feasible  times. 

•  Inputs  from  testing  to  logistic  analysis 
and  evaluation  of  maintainability,  reliability 
and  testability. 


Figure  2-3  shows  the  principal  1TP  activi¬ 
ties  and  precedence  relationships  among 
them.  The  principal  output  of  the  test  plan¬ 
ning  function  is  the  1TPP,  prepared  in  coopera¬ 
tion  with  the  responsible  line  organizations, 
whose  representatives  on  the  1TPB  assure 
free  flow  of  information  among  responsible 
groups.  The  results  of  development  phase 
tests  contribute  to  planning  for  production, 
pre-deployment,  deployment  and  special  oper¬ 
ations  phase  tests.  The  1TPP  evolves  through¬ 
out  the  program  until  it  describes  tests 
planned  for  these  later  program  phases. 

The  ITPB  serves  as  the  overall  administra¬ 
tor  of  the  integrated  test  program.  The  board 
monitors  and  guides  test-related  activities  and 
the  use  of  test  resources  throughout  the 
program.  In  performing  this  function  central¬ 
ly,  the  ITPB  provides  a  forum  for  assembling 
and  evaluating  test  program  objectives,  plans, 
problems,  and  data,  for  resolving  conflicts 
among  schedules,  priorities  and  interpreta¬ 
tions  of  data,  and  for  documenting  and 
reporting  test  program  progress. 

The  ITPB  should  include  a  representative 
from  each  unit  having  significant  test  program 
responsibilities  in  the  contractor’s  organiza¬ 
tion.  The  board  begins  its  work  by  developing 
guidelines  and  instructions,  coordinating,  re¬ 
viewing  and  approving  documents  that  define 
test  requirements,  plans,  procedures,  sched¬ 
ules  and  costs.  Later,  the  board  integrates  and 
optimizes  the  total  test  program,  monitors 
performance  of  the  tests  and  aids  in  solving 
a  variety  of  day-to-day  operating  problems. 

Pass/fail  criteria  are  developed  prior  to 
initiation  of  the  test.  As  tests  are  completed, 
the  ITPB  reviews  the  resulting  data  (some¬ 
times  in  reduced  form),  decides  whether  the 
tested  item  has  passed  or  failed  and,  in  the 
latter  event,  specifies  the  nature  and  extent 
of  corrective  action  or  retesting  necessary 
before  approval  can  be  given.  Review  of  test 
results  is  a  particularly  important  function  of 
the  board  in  qualification  and  requalification 
tests  of  hardware  and  in  validation  and  accep¬ 
tance  tests  of  software,  where  the  collective 
experience  and  judgment  of  the  ITPB  mem¬ 
bers  are  applied  to  support  important,  often 
difficult,  decisions  of  approval  or  rejection 
and  retest. 
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Finally,  the  board  maintains  essential  test- 
related  records  and  at  intervals  provides 
reports  of  the  status  of  the  program  to  con¬ 
tractor  management  and  operating  groups. 
In  discharging  these  responsibilities  during 
the  development  phase,  the  board  helps  to 
generate  information  necessary  for  planning 
tests  to  be  conducted  later  in  production  and 
later  program  phases;  for  example,  the  opti¬ 
mum  frequency  of  production  assessment 
tests,  baseline  parameter  statistics  against 
which  aging  test  results  will  be  measured,  and 
telemetry  data  necessary  to  evaluate  system 
performance. 

When  carried  out  effectively,  the  ITP  con¬ 
cept  confers  a  variety  of  advantages  over 
decentralized  methods  of  test  program 
management.  It  helps  assure  that  necessary 
tests  are  accomplished  and  unnecessary  tests 
avoided.  Optimum  use  is  made  of  facilities 
and  other  test  resources,  resulting  in  reduced 
program  costs.  Communication  is  enhanced 
by  having  responsible  operating  groups 
participate  in  the  planning  and  decision 
processes  by  means  of  the  ITPB.  Data  of  high 
quality  are  acquired,  leading  to  easier  and 
more  accurate  data  analysis,  increased  confi¬ 
dence  in  the  results  of  the  test  program  and 
in  the  capabilities  of  the  developed  product. 

2.3  ITPB  ORGANIZATION  AND  MAJOR 
ACTIVITIES 

The  ITPB,  as  established  by  program  man¬ 
agement,  should  include  a  chairman  and 
supporting  staff,  permanent  members  and,  as 
necessary,  special  members  and  procuring 
activity  representatives. 

The  major  activities  of  the  ITPB  include: 

•  Initiate  analysis  of  test  program  require¬ 
ments  and  constraints. 

•  Evaluate  program  alternatives. 

•  Issue  instructions,  forms,  procedures  and 
schedules  to  govern  preparation  of  individual 
test  plans,  and  the  implementation  and 
reporting  of  the  programs. 

•  Standardize  collection  of  test  data. 

•  Monitor  test  planning  by  line  groups  and 
resolve  questions. 

•  Assure  traceability  of  test  requirements. 

•  Develop  methods  for  assigning  priorities 
to  planned  tests. 


•  Evaluate  planned  tests  and  test-defining 
documents  for  compliance  with  program  re¬ 
quirements. 

•  Evaluate  test  equipment  and  facilities 
and  recommend  optimal  approach  to  re¬ 
sources,  schedules  and  requirements. 

•  Prepare  Integrated  Test  Program  Plan, 
together  with  required  supporting  data. 

•  Review  test  data  as  they  are  evolved,  and 
monitor  the  maintenance  and  storage  of  test 
data. 

•  Verify  compliance  with  test  require¬ 
ments  and  validate  test  results  for  hardware 
and  software  testing. 

•  Administer  changes  to  the  1TPP  within 
applicable  program  constraints  and  authority 
delegated  by  the  program  manager. 

•  Issue  periodic  program  status  reports 
and  program  plan  revisions. 

The  chairman’s  responsibilities  include; 

•  Preside  at  meetings  and  issue  minutes. 

•  Obtain  and  document  board  concurrence 
on  all  test  program  direction  and  subsequent 
changes. 

•  Attempt  to  resolve  differences  among 
board  members  and,  where  differences  cannot 
be  resolved,  present  alternatives  and  recom¬ 
mendations  to  the  program  manager. 

•  Issue  test  program  status  reports  to  the 
program  manager. 

•  Sign  all  documents  requiring  ITPB 
approval  or  concurrence. 

Because  of  the  multiple  interfaces  and 
activities  of  a  test  program,  the  ITPB  should 
include  pennanent  members  who  represent 
the  principal  line  organizations  such  as  design, 
system,  test,  and  product  assurance.  These 
board  members  act  as  liaisons  between  the 
test  program  and  their  respective  organiza¬ 
tions. 

The  permanent  members’  responsibilities 
include: 

•  Integrate  test  program  activities  within 
line  groups  and  act  as  group  representatives 
on  the  board. 

•  Evaluate  planned  tests  for  suitability  as 
sources  of  design,  quality,  reliability,  main¬ 
tainability  and  accuracy  data. 

•  Provide  alternative  solutions  to  test 
program  problems  and  conflicts. 

•  Secure  engineering  analyses  of  failures 
during  tests. 
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In  general,  permanent  members  should  be 
senior  technical  personnel,  knowledgeable 
of  performance  and  test  requirements,  and 
able  to  trade  off  competing  solutions  to  test 
problems. 

Special  representatives  to  the  board  serve 
as  limited  members  on  an  as-needed  basis. 
Special  members  may  include  representatives 
from  various  technologies,  test  laboratories, 
source  inspectors,  suppliers  or  other  specialist 
groups.  Special  members’  responsibilities  are 
to  provide  expert  opinion  and  advice  to  the 
board. 


Establishing  and  managing  an  1TPB  can  be 
more  complicated  when  the  contractor  is  a 
system  design  agency,  and  all  or  part  of  the 
development  engineering  work  is  to  be  done 
by  subcontractors.  In  these  cases  the  contrac¬ 
tor  should  consider  requiring  that  major  sub¬ 
contractors  establish  in-house  test  program 
boards,  so  that  day-to-day  administrative 
decisions  affecting  the  test  program  can  be 
made  at  the  lowest  appropriate  level  of  re¬ 
sponsibility.  Alternatively,  if  subcontractors 
have  limited  design  responsibility,  it  may  be 
preferable  to  have  the  subcontractors  repre¬ 
sented  on  the  system  design  agency’s  ITPB. 
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Section  3 

PREPARATION  FOR  TEST  PLANNING 


3.1  ESTABLISHING  THE  ITPB  AND 
OPERATING  POLICIES 

The  program  manager,  in  agreement  with 
company  line  organization,  decides  early  in 
the  program  what  management  tools  will  be 
used  to  assure  that  the  ITP  is  effectively 
planned  and  implemented.  The  ITPB  should 
be  defined  in  a  company  procedure,  program 
bulletin  or  comparable  document  which  out¬ 
lines  the  board’s  operating  methods  and  the 
responsibilities  of  the  chairman  and  perma¬ 
nent  members.  This  document  should  also 
reflect  the  policy  of  the  company  toward  the 
integrated  test  program  and  the  use  of  the 
ITPB  to  implement  that  policy.  The  policy 
should  establish  the  authority  of  the  ITPB 
for  planning  and  implementation  to  assure  an 
effective  ITP. 

3.2  TEST  PROGRAM  CONSTRAINTS 

Initial  constraints  may  be  imposed  on  the 
test  program  by  the  contract,  by  contractor 
management,  or  by  circumstances.  Contrac¬ 
tual  constraints  may  include  tests  that  are 
mandatory  under  the  contract,  specified  test 
sample  sizes,  specified  test  durations,  as  well 
as  constraints  on  performance,  cost  and 
schedule.  Management  constraints  might  in¬ 
clude  the  level  of  producer’s  risk  acceptable 
in  the  program,  the  extent  of  the  develop¬ 
ment  effort  to  be  undertaken,  and  additional 
restrictions  on  test  costs  and  schedules.  Infor¬ 
mation  on  circumstantial  constraints,  such  as 
limitations  on  the  capacities  or  capabilities  of 
particular  test  facilities,  is  normally  supplied 
by  the  cognizant  line  organizations;  it  is  for 
this  reason  that  effective  information  ex¬ 
change  between  line  organizations  and  the 
ITPB  is  of  paramount  importance  throughout 
the  planning  process. 


Constraints  such  as  those  cited  above  must 
be  identified  and  documented,  because  they 
define  the  bounds  within  which  the  planning 
process  must  operate  to  develop  an  optimum 
test  program.  The  test  program  is  then  de¬ 
fined  by  means  of  analysis  and  planned 
choices  among  discretionary  factors. 

3.3  ITPB  PRE-EVALUATION  PLANNING 
ACTIVITIES 

Before  requesting  individual  test  plans  from 
line  organizations,  the  ITPB  must  be  prepared 
to  evaluate  those  plans.  In  order  to  evaluate 
proposed  tests,  particularly  those  having  veri¬ 
fication  as  a  major  objective,  the  board  must 
have  available  a  complete  description  of  the 
capabilities  required  of  the  product,  since  it 
is  those  capabilities  that  must  be  verified. 

The  general  nature  and  scope  of  test  pro¬ 
grams  must  be  structured  from  the  beginning 
to  meet  the  objectives  of  the  various  contract 
phases  beginning  with  development.  In  de¬ 
velopment,  the  ITPB  monitors  the  test  and 
analysis  effort  to  see  that  all  requirements 
(usually  paragraph  3)  of  the  development 
specification  have  been  verified.  This  is  im¬ 
portant  as  a  contractual  obligation,  and  be¬ 
cause  production  tests  (to  be  described  in 
the  Production  Test  And  Inspection  Plan), 
will  be  based  upon  parameters  verified  in  the 
development  program. 

Because  some  requirements  may  quite 
possibly  be  verifiable  by  means  other  than 
tests  (i.e.,  analysis,  simulation,  pre-existing 
data),  the  overall  requirements  should  be 
identified  by  a  systematic  examination  of  all 
applicable  verification  methods.  In  particular, 
the  board  should  evaluate  the  ability  of  tests 
at  various  assembly  levels  to  provide  meaning¬ 
ful  verification  data.  Here  the  ITPB  looks  for 
the  most  effective  test  program  to  verify  that 
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the  development  specification  is  met  and  that 
production  testing  will  not  degrade  the  de¬ 
veloped  item. 

The  1TPB  review  should  confirm  the  trace- 
ability  of  requirements  and  assure  that  the 
planned  test  program  is  not  unbalanced,  veri¬ 
fying  some  parameters  too  often  and  others 
not  at  all  or  less  often  than  necessary.  This 
activity  should  be  accomplished  by  the  ITPB 
in  direct  discussions  with  the  program  mana¬ 
ger  and  responsible  line  managers.  Out  of  this 
activity  the  board  should  develop  and  docu¬ 
ment  specific  policy  recommendations  to 
promote  uniformity  in  detailed  test  planning 
by  line  groups.  For  example,  policy  state¬ 
ments  or  recommendations  are  desirable  in 
answer  to  test-planning  questions  such  as  the 
following: 

•  When  can  engineering  analysis  or  previ¬ 
ous  data  be  used  in  lieu  of  tests? 

•  What  are  the  preferable  types  of  test  for 
achieving  a  given  test  objective  (e.g.,  if  the 
test  objective  is  to  identify  failure  modes, 
what  types  of  tests  are  preferred)? 

•  What  criteria  should  be  used  in  selecting 
hardware  levels  fortesting  (e.g.,  at  what  assem¬ 
bly  level  will  a  given  type  of  test  be  most 
effective,  least  costly)? 

•  What  approach  should  be  followed  in 
planning  environmental  tests?  Is  the  use  en¬ 
vironment  to  be  simulated  exactly?  Should 
overstress  testing  be  used  and  when?  Should 
environmental  stresses  be  applied  sequentially 
or  simultaneously  and,  if  sequentially,  in  what 
order?  Should  the  use  of  outside  test  facilities 
be  considered? 

•  What  policies  are  appropriate  for  plan¬ 
ning  hazardous  types  of  tests?  Tests  to  failure? 
Reliability  demonstration  tests?  Maintainabili¬ 
ty  demonstration  tests?  Accuracy  tests? 

•  What  should  be  the  extent  of  and  ap¬ 
proach  to  tests  that  are  destructive,  of  long 
duration,  or  very  costly? 

•  When  should  multiple-purpose  tests  be 
considered  (e.g.,  use  of  qualification  test  data 
for  reliability  or  assessment  of  accuracy)? 

•  Can  reliability  and  maintainability  dem¬ 
onstrations  be  completed  during  development 
or  must  they  be  partially  deferred  to  produc¬ 
tion? 

•  Are  data  system  enhancements  needed 
to  support  the  planned  test  program? 


3.4  PROCEDURAL  INSTRUCTIONS 

The  ITPB  should  issue  policies  and  pro¬ 
cedural  instructions  as  necessary  to  facilitate 
adequate  responses  by  line  organizations  in 
the  form  of  individual  test  plans.  Planners 
should  be  apprised  of  the  degree  of  detail 
appropriate  to  various  types  of  test  plans. 
Ideally,  the  board  should  furnish  sample  test 
plan  contents  such  as  those  shown  in  Figures 
4-3  and  4-8. 

The  board  should  confirm  the  existence  of 
uniform  test  data  collection  procedures  that 
satisfy  all  data  needs  of  the  contractor  and 
SSPO.  NAVSEA  OD  29304B  provides  guide¬ 
lines  in  these  areas.  The  defining  documents 
should  be  identified  and  included  by  refer¬ 
ence  in  individual  test  plans.  If  it  is  deemed 
desirable  to  relax  data  collection  procedures 
for  development  tests,  minimum  data  require¬ 
ments  for  those  tests  should  be  defined. 

The  board  should  describe  its  intended 
operating  policies,  procedures  and  require¬ 
ments  for  implementing  the  test  program.  It 
should  inform  line  organizations  of  test¬ 
defining  documents  the  board  must  review 
and  approve,  criteria  for  deviations,  and  rules 
of  test  program  operation,  including  the  lati¬ 
tude  of  discretionary  authority  that  will  be 
granted  to  test  directors  at  various  sites.  The 
board  should  define  the  standards  it  will 
apply  to  evaluate  the  validity  of  tests  and  the 
effectiveness  of  corrective  actions  following 
test  failures. 

Guidance  for  budgetary  planning,  support¬ 
ing  estimated  costs,  preparing  schedules  and 
establishing  program  milestones  should  also 
be  issued,  and  the  board  should  schedule  the 
initial  submission  of  individual  test  plans  by 
line  organizations. 

3.5  ITPB  OPERATION 

The  ITPB’s  ability  to  operate  depends  on 
a  continuous  flow  of  information.  For  maxi¬ 
mum  efficiency  of  operation,  the  ITPB  should 
have  means  of  standardizing  and  utilizing  the 
large  amount  of  information  a  weapon  system 
program  can  generate.  The  following  are  gen¬ 
eral  examples  of  documents  the  ITPB  can  use 
for  these  purposes: 
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•  Verification  Allocation  Matrix 

•  Master  Test  Planning  Summary  Chart 

•  Master  Test  Schedule 

•  Test  Milestone  Chart 

•  Test  Status  Log 

•  Integrated  Test  Program  Status  Report 

•  Test  Planning  Information  Form 

•  Test  Program  Change  Proposal 

In  addition  to  the  information  contained 
in  the  documents  noted  above,  the  ITPB  re¬ 
views  the  individual  item  Test  Plans  and  Test 
Procedures.  With  this  complete  package  the 
ITPB  can  evaluate  the  necessity  and  suffi¬ 
ciency  of  the  proposed  tests,  to  structure  a 
balanced  and  efficient  program  and  to  define 
the  scope  of  tests  needed  in  production  and 
later  program  phases. 

3.5.1  Verification  Allocation  Matrix 

An  init.  i  step  in  setting-up  an  integrated 
test  program  is  to  evaluate  the  item  perfor¬ 
mance  requirements.  Each  performance  re¬ 
quirement,  as  delineated  in  the  overall  item 
specification  or  sub-tier  specification,  should 
be  analyzed  to  determine  the  best  method  of 
verification  and  the  test  phase  in  which  verifi¬ 
cation  must  be  accomplished.  Figure  3-1  pro¬ 
vides  an  example  of  a  verification  allocation 
matrix.  The  board  reviews  the  matrix  for 
completeness,  makes  or  requests  corrections 
and  assigns  a  test  number  to  each  test.  For 
complex  programs  a-  sequential  test  number¬ 
ing  system  can  be  established  with  an  identi¬ 
fiable  code  for  each  area  of  design. 

The  ITPB  should  review  all  matrices  to 
detect  overlaps,  voids  and  redundancies  and 
to  make  initial  program  decisions  to  combine, 
add  or  eliminate  tests.  Later,  as  information 
is  received  from  design  and  test  groups,  the 
program  will  be  reviewed  and  redirected  as 
necessary.  When  the  initial  matrices  have  been 
evaluated,  the  board  can  initiate  its  status 
information  system. 

3.5.2  Master  Test  Planning  Summary  Chart 

The  Master  Test  Planning  Summary  Chart 
(Figure  3-2)  is  a  key  element  in  detecting  test 
program  voids,  overlaps  and  redundancies. 
Column  1  of  the  chart  can  be  annotated  with 
the  test  numbers  assigned  to  the  tests  on  the 


verification  ailoc,'*’on  matrix.  The  informa¬ 
tion  to  be  completed  the  Master  Test  Plan 
Summary  Chart  is  self  explanatory  and  is 
extracted  from  test  planning  information 
forms  discussed  in  Section  4. 

3.5.3  Master  Test  Schedule  And  Test 
Milestone  Chart 

The  Master  Test  Schedule  provides  a  chron¬ 
ological  listing  of  all  planned  test  dates 
referenced  to  the  Work  Breakdown  Structure 
(WBS)  or  drawing  tree.  The  ITPB  staff  can 
utilize  the  schedule  to  monitor  test  progress 
and  assure  that  all  planned  tests  are  per¬ 
formed.  Appropriate  annotations  are  made 
on  the  schedule  to  indicate  test  completion, 
deferral,  rescheduling,  success  or  failure.  The 
schedule  should  be  capable  of  highlighting 
test  item  problems  and  test  program  prob¬ 
lems. 

The  Test  Milestone  Chart  is  a  pictorial  rep¬ 
resentation  of  the  test  program  which  indi¬ 
cates  the  interrelationships  of  all  tests.  The 
chart  should  include  all  the  ITP  tests  and  be 
based  upon  program  or  contract  milestones. 
The  primary  purpose  of  the  milestone  chart 
is  to  be  able  to  assess  the  dependencies  of  the 
test  program  when  test  slippage  and  test 
facility  downtimes  occur  and  tests  must  be 
rescheduled,  or  when  tests  must  be  added. 
Once  the  milestone  chart  has  been  drawn, 
necessary  adjustments  can  be  done  by  attach¬ 
ing  overlays  to  the  original  chart. 

3.5.4  Test  Status  Log 

The  ITPB  chairman  should  maintain  a 
current  overall  Test  Status  Log.  Normally,  the 
ITPB  staff  provides  the  chairman  with  daily 
information  on  in-process  testing,  test  prog¬ 
ress,  failures,  test  facility  problems,  etc.  to 
maintain  the  currency  of  the  log. 

3.5  5  Status  Reports 

The  ITPB  provides  status  reports  on  a 
periodic  basis,  usually  quarterly,  to  formally 
apprise  the  program  manager  of  test  status. 
These  reports  form  the  basis  for  Integrated 
Test  Program  Status  Reports  which  are  sub 
mitted  to  SSPO.  Figure  3-3  shows  the  content 
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of  Verification  Allocation  Matrix  with  Assignment  of  Test  Numbers 
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of  a  status  report,  with  the  required  content 
of  the  NAVSEA  OD21549A  Integrated  Test 
Program  Status  Report  indicated. 

3.5.6  Forward  Planning  For  Testing 
In  Later  Program  Phases 

One  of  the  duties  of  the  ITPB  in  the  de¬ 
velopment  phase  is  to  provide  forward  plan¬ 
ning  for  testing  that  will  be  required  in 
production  and  later  program  phases.  Proper 
planning  during  the  development  phase  will 
provide  a  smooth  transition  to  later  testing 
(predeployment,  deployment,  etc.).  The  board 
should  identify  tests  required  in  production, 
together  with  all  pertinent  information  de¬ 
veloped  in  the  ITP,  such  as  test  parameters, 
test  results,  test  data  and  test  procedures, 
which  can  be  utilized  to  plan  production 
phase  tests.  Later,  as  the  program  progresses, 
the  ITPB  will  identify  updated  ITP  informa¬ 
tion  that  can  be  used  to  support  planning  for 
post-production  tests. 

3.5.7  Indentured  System  Breakdown 

To  assure  that  all  items  to  be  tested  are 
accounted  for  in  the  integrated  test  program, 
the  ITPB  needs  a  complete  listing  of  system 
items.  The  listing  should  be  by  part  number, 
assembly  number  or  software  module  and 
higher  assembly  numbers,  so  that  the  ITPB 
can  compare  the  listing  against  the  Master 
Test  Planning  Summary  Chart.  The  board 
should  be  on  distribution  for  all  subsequent 
updates  of  the  breakdown,  which  should  also 
identify  design  engineers  responsible  for  each 
item. 

3.6  TEST  PROGRAM  PHASES  AND 
OBJECTIVES 

An  SSPO  system  undergoes  several  cate¬ 
gories  of  tests  during  phases  of  its  life  cycle. 
Development  phase  tests  support  design  re¬ 
lease,  evaluate  the  limits  of  system  perfor¬ 
mance,  and  the  adequacy  of  system  design 
and  associated  production  processes.  Tests  in 
the  production  phase  insure  that  production 
disciplines  are  maintained  and  that  RM&Q 
are  not  degraded  during  production.  Pre¬ 
deployment  tests  determine  the  readiness  of 
the  complete  weapon  system  for  operational 
deployment.  Tests  in  the  deployment  phases 


CONTENT  OF  ITP  STATUS  REPORT 

*1.  Description  of  significant  problems,  if  any,  re¬ 
medial  actions  taken  and  schedules  for  accomplish 
ing  planned  action. 

*2.  Test  schedules  with  current  updates 

*3.  Listing  of  all  planned  tests  completed  during  the 
reporting  period,  with  indication  of  whether  test 
objectives  were  met. 

*4.  A  description  and  status  of  all  failures  that  oc¬ 
curred  during  the  reporting  period. 

*5.  Status  of  open  failures  and  description  of  correc¬ 
tive  actions  taken  on  failures  closed  during  the 
reporting  period. 

6.  Status  of  planned  capital  test  facility/equipment 
procurement. 

7.  Status  of  test  facility  uptime. 

8.  Status  of  testing  at  remote  and  outside  facilities. 

9.  Status  of  GFM. 


•Required  by  NAVSEA  OD21549A  for  the  Integrated 
Test  Program  Status  Report. 

Figure  3-3.  Typical  Content  Of  ITP  Status  Report 

verify  weapon  system  performance  under 
mission  conditions,  and  assess  RM&Q  during 
operational  life. 

In  order  to  assure  uniform  terminology  for 
classifying  proposed  tests,  the  contractor 
should  standardize  the  test  nomenclature  to 
be  used  in  each  program  phase.  The  contrac¬ 
tor  is  encouraged  to  use  his  own  terminology 
to  identify  tests,  but  these  tests  should  in¬ 
clude  the  general  test  categories  of  NAVSEA 
OD21549A,  such  as  development,  qualifica¬ 
tion,  software  validation,  demonstration,  and 
production  assessment  tests.  Figure  3-4  shows 
the  general  flow  of  testing  activities  through 
an  acquisition  program. 

3.6.1  Development  Phase  Tests 

The  development  phase  test  program  con¬ 
sists  of  development,  engineering  evaluation, 
qualification  and/or  requalification,  software 
validation  and  acceptance  testing.  (Develop¬ 
ment  tests  within  this  section  refer  to  a 
specific  class  of  tests,  not  the  entire  develop¬ 
ment  phase  test  program). 

In  general,  later  categories  of  tests  are  more 
formal  than  earlier  categories.  Development 
tests  may  be  conducted  on  pre-prototype 
equipment,  such  as  breadboard  circuits  or 
models,  or  on  parts  not  yet  committed  for  use 
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in  the  system.  Development  tests  may  also  be 
applied  in  areas  of  design  requiring  special 
evaluation.  (One  objective  of  Failure  Mode, 
Effects  and  Criticality  Analysis  is  to  identify 
such  areas).  Engineering  evaluation  tests  are 
generally  conducted  on  prototype  and  pre- 
production  equipment.  Qualification  tests  are 
conducted  on  prime  hardware,  but  may  be 
conducted  on  pre-production  hardware  con¬ 
forming  to  the  tactical  design  and  manu¬ 
factured  under  all  applicable  production 
methods  and  controls. 

While  NAVSEA  OD21549A  generally  ad¬ 
dresses  tactical  equipment  and  tactical  support 
equipment,  the  contractor  may  want  to  con¬ 
sider  including  in  the  ITP  support  items,  such 
as  data  reduction  software  and  simulation 
equipments,  that  are  being  developed. 

3.6.1 .1  Development  Tests 

Development  tests  are  investigatory  and  are 
intended  to  develop  concepts,  select  design 
solutions,  assess  the  capabilities  of  vendors, 
select  candidate  materials  and  establish  or 
verify  design  parameters  and  specification  re¬ 
quirements.  Data  from  development  tests  are 
evaluated,  to  determine  whether  an  accept¬ 
able  design  solution  has  been  reached.  Thus, 
development  tests  are  part  of  the  design 
activity.  They  are  performed  on  models, 
breadboard  circuits,  parts,  components  or 
other  elements  of  the  total  product. 

Development  tests  differ  from  tests  in¬ 
tended  for  performance  verification  in  several 
important  ways.  Generally,  development  tests 
are  relatively  unstructured  and  unconstrained 
by  the  contract.  This  is  a  consequence  of  the 
exploratory  element  common  to  such  tests. 
Usually,  there  is  a  proper  emphasis  on  flexible 
and  informal  procedures  planned,  imple¬ 
mented  and  evaluated  by  highly  trained  en¬ 
gineering  personnel. 

The  principal  questions  that  confront  the 
1TPB  when  it  evaluates  development  tests 
typically  concern  the  need  for  the  proposed 
tests  rather  than  details  of  their  performance. 
Therefore,  it  is  important  that  a  development 
test  plan  answer  the  question  of  need.  It 
should  cover  the  intended  use  of  the  informa¬ 
tion  to  be  gained  from  the  tests.  And  it  should 
assure  effective  feedback  of  the  information 
to  engineering  tasks  such  as  the  setting  of 
specification  limits.  If  reasonable  alternatives 


to  the  proposed  approach  exist,  or  if  it  is 
possible  to  combine  the  proposed  tests  with 
others,  those  options  should  be  evaluated  lor 
cost  and  effectiveness.  In  instances  when  de¬ 
velopment  tests  are  to  be  utilized  to  evaluate 
or  verify  design  capability,  the  1TPB  should 
require  more  formal  controls  and  documenta¬ 
tion. 

3. 6. 1.2  Engineering  Evaluation  Tests 

Engineering  Evaluation  Tests  (EET)  pro¬ 
vide  data  necessary  to  verify  that  the  design 
solutions  selected  during  development  can 
meet  specified  functional  and  environmental 
requirements  when  the  item  is  fabricated 
under  normal  manufacturing  processes  and 
controls.  They  are  performed  at  the  highest 
assembly  levels  practicable,  on  prototype  and 
pre-production  items  representing  intended 
production  as  closely  as  possible.  EETs  are 
used  to  assess  the  degree  to  which  the  item 
meets  design  intentions,  to  determine  the 
effects  of  varying  stress  levels  or  combina¬ 
tions  and  sequences  of  environments,  and  to 
identify  failure  modes  and  trends.  Data  from 
engineering  evaluation  and  similar  verification 
tests  are  used  to  make  the  milestone  decisions 
that  the  capabilities  of  items  tested  have  been 
successfully  verified  and  that  their  designs  are 
acceptable  for  production  release.  An  aging 
test  program  (see  Section  3. 6. 2. 5)  may  be 
begun  as  part  of  engineering  evaluation,  to 
evaluate  the  effects  of  selected  long-term 
environments. 

Engineering  evaluation  tests  must  be 
planned  in  greater  detail  than  development 
tests.  Questions  facing  the  ITPB  in  its  review¬ 
ing  capacity  concern  both  the  need  for  and 
implementation  of  the  proposed  tests.  A 
variety  of  questions  regarding  implementation 
are  of  particular  interest  in  EETs  and  should 
be  answered  fully  by  the  plans  and  procedures 
submitted  to  the  ITPB:  Is  the  test  hardware  as 
nearly  prime  as  practicable?  How  significant 
are  the  remaining  differences?  Can  the  tests 
be  conducted  at  a  higher  assembly  level  with¬ 
out  excessive  loss  of  sensitivity?  Do  the 
planned  environmental  stress  profiles  and 
sequences  relate  meaningfully  to  mission  re¬ 
quirements?  Will  the  item  be  operated  during 
or  after  environmental  exposure  and  will  the 
test  exercise  the  item  fully? 
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EETs  may  provide  data  for  assessing  the 
accuracy  of  a  tested  item  (i.e.,  its  contribu¬ 
tion  to  equipment,  subsystem  and  weapon 
system  error).  When  accuracy  is  to  be  evalu¬ 
ated,  the  ITPB  should  verify  that  engineering 
evaluation  test  plans  define  the  data  neces¬ 
sary'  to  solve  the  mathematical  model  that  is 
to  be  used  to  estimate  error  statistics,  and 
that  the  tests  are  structured  to  obtain  the 
data. 

EETs  may  also  provide  data  for  the  assess¬ 
ment  of  reliability  and  maintainability.  When 
utilizing  this  data  for  assessment,  the  validity 
of  the  data  should  be  evaluated  by  the  re¬ 
liability  and  maintainability  line  organiza¬ 
tions,  in  accordance  with  criteria  given  in 
NAVSEA  OD29304B.  Test  planners  should 
be  conscious  of  the  opportunities  presented 
by  EET  for  extending  knowledge  of  the  hard¬ 
ware  by  applying  stress  levels  that  exceed 
design  requirements,  while  at  the  same  time 
confirming  the  item’s  suitability  for  program 
use. 

An  environmental  design  requirement  may 
be  specified  as  the  maximum  mission  environ¬ 
ment  plus  a  specified  margin.  Often,  however, 
specifications  must  be  issued  before  a  proper 
margin  has  been  defined.  Environmental  ex¬ 
tended-limit  tests  may  be  performed  during 
development  on  items  for  which  the  weapon 
specification  does  not  include  appropriate 
design  margins.  The  tests  are  conducted  at 
levels  that  exceed  the  mission  environment. 
The  levels  are  usually  increased  progressively 
until  failure  occurs,  in  order  to  determine  the 
limits  of  the  design.  However,  data  from  over¬ 
stress  tests  (non-mission  environments  or  ex¬ 
tended  stress  environments)  should  not  be 
used  for  reliability  measurement. 

Definitions  of  failure  may  demand  special 
attention  when  planning  EETs.  Sometimes 
the  limits  of  satisfactory  component  per¬ 
formance  in  a  system  are  unknown  or  diffi¬ 
cult  to  determine,  or  the  specified  limits  may 
reflect  accepted  standards  of  manufacturing 
quality  rather  than  actual  needs  of  the  system 
(e.g.,  the  common  practice  of  specifying  insu¬ 
lation  resistance  in  the  megohm  region),  or 
the  item  under  test  may  have  been  manufac¬ 
tured  by  other  than  usual  production  tech¬ 
niques.  Therefore,  in  determining  whether  or 
not  a  failure  in  EET  is  relevant,  it  may  be 


meaningful  to  consider  not  only  the  cause  of 
the  failure,  but  also  whether  the  observed 
anomaly  would  constitute  a  failure  of  the 
item  if  it  were  to  occur  in  service. 

3.6. 1.3  Qualification  Testing 

Qualification  tests  are  conducted  to  demon¬ 
strate  the  ability  of  prime  hardware  to  meet 
all  specified  functional,  environmental,  and 
related  requirements.  Qualification  tests  are 
performed  on  prime  items  manufactured  in 
conformity  with  all  production  processes  and 
quality  controls.  They  usually  include  perfor¬ 
mance  under  simulated  mission  environments 
and  use  conditions,  as  identified  in  the  item 
specification.  They  are  formal  tests  and  must 
be  completely  documented.  Access  to  qualifi¬ 
cation  test  areas  should  be  controlled  to  the 
extent  necessary  to  assure  proper  execution 
and  integrity  of  the  tests.  The  ITPB  should 
closely  monitor  the  progress  of  qualification 
tests.  Data  from  these  tests  are  used  to  sub¬ 
stantiate  compliance  with  specifications  be¬ 
fore  first  delivery,  so  that  changes  can  be 
made  if  necessary  without  adversely  affecting 
production  flow.  The  ITPB  should  assure  that 
if  an  item  cannot  pass  qualification,  failure 
analysis  and  corrective  action  are  initiated 
immediately.  Qualification  data  may  also  pro¬ 
vide  estimates  of  baseline  parameter  distri¬ 
butions,  which  can  be  compared  to  future 
aging  test  data  for  use  in  detecting  age  degra¬ 
dation. 

NAVSEA  OD21549A  requires  the  contrac¬ 
tor  to  develop  a  Qualification  Program  Plan 
(QPP),  which  defines  the  program  that  is 
established  and  maintained  to  assure  the  capa¬ 
bility  of  items  to  meet  specification  require¬ 
ments.  Figure  3-5  lists  the  content  of  a  QPP. 

The  ITPB  should  assure  that  qualification 
test  plans  prepared  by  line  organizations  re¬ 
flect  compliance  with  the  contract  require¬ 
ments  and  the  approved  QPP.  The  ITPB 
should  have  approval  authority  over  qualifi¬ 
cation  test  plans.  The  ITPB  should  evaluate 
any  proposed  modifications  or  deviations 
from  specified  requirements,  whether  delin¬ 
eated  in  the  plans  or  separately  proposed, 
prior  to  submittal  to  the  procuring  activity 
for  approval.  The  ITPB  may  recommend  that 
certain  qualification  tests  be  omitted,  if  it 
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believes  an  item  does  not  require  testing  based 
on  the  item  meeting  the  “qualification  by 
similarity”  criteria  of  NAVSEA  OD21 549A. 
The  ITPB  should  exercise  caution  when 
recommending  an  item  for  deletion  from 
qualification  testing,  due  to  the  possible  cost 
and  schedule  impact  of  having  to  restore  to 
the  program  an  item  which  is  later  found  not 
to  meet  the  similarity  criteria.  The  ITPB 
should  also  be  alert  to  the  possibility  that  a 
previously  qualified  item  may  require  requali¬ 
fication  in  accordance  with  the  criteria  in 
NAVSEA  OD21549A  as,  for  example,  when 
the  item  is  to  be  acquired  from  a  second 
source.  When  requalification  is  required,  the 
ITPB  should  alert  the  cognizant  line  organ¬ 
ization  to  begin  the  process  of  defining  and 
scheduling  the  tests. 

The  ITPB  should  review  qualification  test 
procedures,  comparing  them  with  specifica¬ 
tions  and  contractual  requirements  docu¬ 
ments  for  consistency;  any  discrepancies 
should  be  resolved  with  the  line  organization 
and  test  facilities  personnel.  Upon  approval, 
the  ITPB  chairman  should  sign  or  initial  the 
approval  sheets  of  the  procedures. 

Qualification  test  reports  should  also  be 
reviewed  by  the  ITPB  to  assure  that  the  tests 
have  been  conducted  in  complete  compliance 
with  the  approved  procedures. 

3.6. 1.4  Software  Validation  Testing 

Software  Validation  Testing  consists  of 
module,  module  integration  and  software 
system  tests,  to  assess  the  degree  to  which 
software  modules  and  systems  meet  design 
requirements. 

A  module  is  a  functional  unit  of  software, 
generally  between  100  and  200  executable 
statements.  Software  tests  are  performed  on 
successively  larger  units  of  software,  (i.e., 
modules,  combinations  of  two  modules, 
combinations  of  these  two  modules  with  a 
third  module,  etc.)  leading  ultimately  to  test¬ 
ing  of  the  complete  software  system. 

There  are  several  approaches  to  sequencing 
the  testing  and  merging  of  modules  into  larger 
entities  (Figure  3-6).  Three  of  the  most  suc¬ 
cessful  approaches  are  designated  as  Bottom- 
Up,  Top-Down  and  Modified  Top-Down. 

In  the  Bottom-Up  approach  the  program 
is  merged  and  tested  from  the  bottom  to  the 


CONTENT  OF  A  QUALIFICATION 
PROGRAM  PLAN 

1.  Describe  the  organization  and  management  of  the 
qualification  program  (eg.,  applicable  policy  state¬ 
ments,  management  directive,  etc  ). 

2.  Describe  the  criteria  used  to  establish  qualification 
test  requirements,  test  locations  and  criteria  for 
deciding  when  an  item  has  been  successfully  quali¬ 
fied. 

3.  Describe  the  qualification  status  of  items  includ¬ 
ing:  reference  documentation  and  qualification 
methods  for  items  that  require  initial  qualification, 
items  considered  qualified  by  virtue  of  previous 
qualification,  including  justification;  and  reasons 
and  extent  of  required  testing  for  items  not  con 
sidered  qualified. 

4.  Include  schedules  for  preparing  qualification  test 
plans  and  procedures  for  each  hardware  item. 

5.  Delineate  the  criteria  and  method  for  revising  and 
resubmitting  the  QPP  to  SSPO. 

6.  Delineate  the  criteria  and  methods  to  be  followed 
in  making  decisions  concerning  the  need  for  re 
qualification. 


Figure  3-5.  Conlent  Of  A  Qualification  Program  Plan 


top.  The  lowest  level  modules  of  the  software 
hierarchy  are  tested  and  merged  with  the  next 
higher  level.  This  process  is  repeated  until  the 
top  level  is  reached.  Since  Bottom-Up  testing 
is  initiated  at  the  lower  levels  of  the  program, 
it  requires  module  drivers.  A  module  driver 
is  a  method  of  feeding  test  case  inputs  to  the 
interface  of  the  module  under  test.  Bottom- 
Up  testing  is  most  useful  when  the  modules 
at  the  bottom  of  the  hierarchy  are  critically 
important,  as  in  many  real-time  systems. 

The  Top-Down  approach  to  software  design 
and  testing  is  initiated  at  the  highest  hier¬ 
archical  level  of  the  program  and  proceeds 
to  the  lowest  level.  The  components  of  each 
hierarchical  level  are  merged  and  tested  with 
the  components  of  the  next  lower  level.  Only 
the  top  level  of  the  program  is  tested  in  isola¬ 
tion.  The  Top-Down  approach  requires  the 
use  of  stubs,  since  the  module  under  test  may 
need  to  pass  information  to  a  lower  hier¬ 
archical  level  not  present  in  the  test  configu¬ 
ration.  In  general,  stubs  can  be  less  complex 
than  drivers. 

In  the  modified  Top-Down  testing  method, 
each  module  to  be  incorporated  into  the  hier¬ 
archy  is  tested  in  isolation  prior  to  its  integra¬ 
tion  into  the  Top-Down  testing  scheme.  This 
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Figure  3-6.  Representative  Hierarchal  Software  Structure 


method  requires  module  drivers  and  stubs  for 
each  module. 

Software  validation  tests  are  generally 
divided  into  three  types:  functional,  stress 
and  fault  tests.  Functional  tests  are  performed 
to  assure  that  modules  satisfy  performance 
requirements,  that  interfaces  between  mod¬ 
ules  are  error-free,  and  that  the  software 
system  performs  all  its  functions.  Stress  test¬ 
ing  is  used  to  test  the  modules  and  system  by 
imposing  stresses  (e.g.,  illegal  inputs,  data 
rates  at  or  exceeding  the  design  limits)  to  test 
the  reaction  of  the  software.  Fault  testing 
assures  that  self-checking,  correcting,  perfor¬ 
mance  monitoring  and  diagnostic  design 
requirements  are  satisfied.  These  tests  as¬ 
sure  that  hardware-to-software,  software-to- 
hardware  and  software-to-software  interfaces 
are  compatible.  As  in  hardware  testing,  use  of 
an  independent  test  team  can  improve  the 
effectiveness  of  software  validation  tests. 

Firmware  is  software  residing  in  (burn¬ 
ed  into)  programmable-read-only  memories 
(PROMs),  read-only  memories  (ROMs),  etc. 
Software  developed  for  delivery  as  firmware 
should  be  tested  prior  to  bum-in  of  PROMs. 
This  testing  is  no  different  from  other  soft¬ 
ware  validation  testing.  After  bum-in,  a  veri¬ 
fication  test  should  be  performed  to  verify 
that  the  bum-in  has  been  done  correcfly.  As 
a  minimum,  program  listings  derived  from  the 
bumed-in  PROMs  should  be  compared  with 
the  pre-bum-in  listings. 


3.6.1 .5  Software  Acceptance  Testing 

Software  acceptance  testing  consists  of 
tests  at  the  highest  level  of  integration  prac¬ 
ticable,  consistent  with  contract  and  delivery 
definitions,  to  demonstrate  that  design  speci¬ 
fications  have  been  met.  These  tests  are  per¬ 
formed  utilizing  intended  system  hardware 
and  support  software  (compilers,  assemblers, 
etc.)  under  actual  operating  conditions. 
“Intended  system  hardware”  and  “actual 
operating  conditions”,  as  defined  by  the 
SSPO  subsystem  concept,  should,  as  a  mini¬ 
mum,  be  accomplished  within  the  subsystem; 
however,  testing  between  subsystems  should 
be  as  defined  in  the  contract.  Again,  these 
tests  are  generally  divided  into  three  areas: 
functional,  stress  and  fault.  Acceptance  tests 
demonstrate  all  man-machine  interfaces  and 
verify  that  documentation  is  satisfactory.  Test 
effectiveness  may  be  improved  by  using  an 
independent  test  team  to  perform  the  accep¬ 
tance  tests.  Software  acceptance  tests  are 
also  run  in  later  program  phases,  whenever 
software  modifications  are  made. 


3.6.1.6  Demonstration  Testing 

Demonstration  testing  as  defined  in 
NAVSEA  OD21549A  pertains  to  reliability 
and  maintainability.  These  tests  are  formal 
tests  and  are  performed  as  required  by  con- 
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tract.  The  purpose  of  demonstration  is  to 
provide  evidence  that  the  item  meets  con¬ 
tractually  specified  reliability  and  maintain¬ 
ability.  Demonstration  should  be  performed 
at  the  highest  assembly  level  practicable  and 
utilizing  items  that  represent  the  production 
configuration.  Considerations  for  planning 
reliability  and  maintainability  demonstrations 
are  presented  in  Section  4.  Guidance  for  plan¬ 
ning  reliability  and  maintainability  demon¬ 
stration  tests  can  be  found  in  NAVSEA 
OD  29304B  and  MIL-STD-471A,  respectively. 

3.6.2  Production  Phase  Tests 

Test  programs  during  production  are  per¬ 
formed  under  controlled  production  condi¬ 
tions  on  a  continuing  basis.  They  are  intended 
to  assure  quality  by  verifying  the  accept¬ 
ability  of  delivered  hardware,  therefore  they 
are  conducted  on  prime  hardware  suitable 
for  service  use.  Data  from  production  tests  are 
used  to  confirm  that  the  items  being  pro¬ 
duced  continue  to  conform  to  the  standards 
established  by  the  design  disclosure  docu¬ 
mentation.  Inherent  in  the  production  test 
planning  process  is  the  need  for  a  smooth 
transition  from  development  to  production. 

3.6.2. 1  Technical  Documentation 
Verification  Tests 

Testing  to  verify  operational  technical 
documents,  including  manuals  and  proce¬ 
dures  for  installation,  operation,  diagnosis 
and  maintenance,  which  are  delivered  under 
contract,  should  be  covered  by  the  integrated 
test  program  plan.  The  responsible  line  organ¬ 
izations  should  plan  in-process  reviews  of  such 
documents  during  their  production,  to  detect 
editorial  or  technical  problems  or  deficiencies, 
and  to  verify  corrective  actions.  Readability 
reviews  are  conducted  on  completed  docu¬ 
ments.  Procedures  presented  in  the  docu¬ 
ments  are  tested  under  operational  condi¬ 
tions,  to  demonstrate  their  accuracy,  com¬ 
pleteness  and  adequacy. 


3. 6. 2. 2  Lowest-Level  Item  Tests 

Production  testing  of  items  .it  me  lowest 
indenture  level  includes  both  acceptance  and 
assessment  tests.  These  items  are  functionally 
tested  with  environmental  conditions  applied, 
to  verify  acceptability  of  functional  perfor¬ 
mance.  The  tests  are  also  designed  to  detect 
subtle  construction,  manufacturing  and  proc¬ 
ess  changes.  Where  functional  tests  are  de¬ 
structive  to  an  item,  such  as  propulsion  or 
pyrotechnic  items,  a  combination  of  non¬ 
destructive  testing  and  functional  testing 
should  be  performed. 

Acceptance  tests  usually  take  place  at  the 
source  and  are  run  by  the  supplier.  Assess¬ 
ment  tests  are  performed  by  the  contractor 
to  verify  the  supplier’s  data,  by  repeating 
measurements  on  the  same  sample  of  items, 
supplemented  by  data  from  additional  items 
accepted  by  receiving  inspection  based  on 
incoming  acceptance  tests. 

A  small  number  of  item  samples,  as  deter¬ 
mined  by  shipping  lot  size  or  number  of 
sublots,  is  subjected  to  visual  and  physical 
analysis  to  detect  construction,  workmanship, 
material  or  process  changes  that  might  de¬ 
grade  part  quality  or  reliability.  Physical 
analysis  should  include  dissection  and  inspec¬ 
tion  by  scanning  electron  microscope. 

Non-destructive  tests  are  quality  tests  that 
apply  a  variety  of  chemical,  magnetic,  sonic, 
radiographic  and  other  evaluation  techniques 
to  the  selected  items  to  detect  degradation, 
while  maintaining  the  performance  integrity 
of  the  items.  Non-destructive  testing  is  usually 
performed  on  100  percent  of  items,  where 
functional  testing  would  be  destructive.  Func¬ 
tional  testing  of  such  items  is  done  on  a 
sampling  basis.  Environmental  conditions  in¬ 
clude  temperature,  vibration,  shock,  humid¬ 
ity,  electromagnetic,  etc.,  which  establish  the 
suitability  of  the  item  for  use.  Vulnerability 
“nuclear)  testing  in  reactors  and  surface 
effects  environments  are  performed  on 
selected  items  (such  as  semiconductors), 
based  on  mission  analysis,  to  assure  that  the 
items  meet  degraded  characteristics  require¬ 
ments. 
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3.6.2.3  Production  Acceptance  Tests 

Production  acceptance  tests  are  performed 
on  end  items,  including  spares,  and  on  lower 
level  items  intended  for  use  in  end  items,  to 
verify  compliance  to  specification  require¬ 
ments  and  to  weed  out  latent  defects.  Accep¬ 
tance  tests  should  reflect  performance  re¬ 
quirements  and  tolerance  limits  defined 
during  development  phase  testing.  These  tests 
should  be  under  conditions  that  simulate  end 
use  of  the  item  to  the  highest  degree  practi¬ 
cable. 

Production  acceptance  tests  are  applied  to 
100  percent  of  production  items  which  can 
be  tested  non-destructively;  most  screening 
and  bum-in  tests  fall  into  this  category.  Where 
test  changes  are  needed  or  tests  need  to  be 
developed,  the  tests  should  be  designed  based 
on  the  criticality  of  the  item  and  to  assure 
that  the  impact  of  undetected  faults  is  accep¬ 
tably  small.  As  a  minimum,  the  test  should 
expose  faults  that  could  materially  reduce 
mission  capability.  Beyond  that,  it  may  be 
necessary  to  test  for  less  severe  faults  when 
warranted  by  their  cumulative  effect  on  sys¬ 
tem  readiness  and  logistic  cost.  The  extent  of 
tests  to  reduce  less  severe  faults  should  be  a 
trade-off  against  incentives,  or  the  cost  of 
testing  against  the  cost  and  manpower  re¬ 
quired  to  correct  failures  later  in  service.  The 
planning  process  for  these  tests  utilizes  the 
information  and  results  of  the  development 
phase  integrated  test  program. 

3.6.2.4  Production  Assessment  Tests 

Production  Assessment  Tests  (PAT)  assess 
the  ability  of  production  controls  to  preclude 
problems  having  potential  impact  on  the 
quality,  safety  and  reliability  of  hardware 
delivered  to  the  fleet.  Periodic  tests  and  in¬ 
spections  are  conducted  on  random  samples 
of  selected  functional  and  structural  items 
that  have  already  passed  acceptance  require¬ 
ments.  The  nature  of  the  tests,  number  of  test 
samples  selected  for  each  assessment,  and 
frequency  of  the  tests  should  be  in  accor¬ 
dance  with  approved  plans  compatible  with 
the  complexity  of  the  production  process  and 
its  controls.  Production  assessment  tests  apply 


a  range  of  functional  and  environmental  tests 
(some  at  degrading  levels)  to  random  samples 
of  production  items.  If  an  item  is  produced 
on  more  than  one  production  line,  or  pro¬ 
cured  from  more  than  one  source,  sample 
selection  should  include  all  lines  and  sources. 
Typically,  the  tests  are  run  on  a  lot  basis,  or 
quarterly  when  production  is  continuous  and 
not  readily  divided  into  lots.  PAT  tests  are 
similar  to  those  used  originally  to  qualify  the 
design  and  manufacturing  process. 

3.6. 2. 5  Aging  Test  Program 

An  aging  test  program  is  a  surveillance  pro¬ 
gram  which  evaluates  the  effects  of  selected 
long  term  (real  time)  environmental  stresses 
on  items  in  use  or  storage.  An  aging  test  pro¬ 
gram,  if  initiated  early  in  the  development 
program,  can  indicate  parameter  degradation 
rates  prior  to  system  deployment.  Aging  tests 
are  intended  to  make  visible  any  degradation 
due  to  service  environments,  handling  and  re¬ 
peated  operation.  An  aging  test  program  plan 
should  include  the  approaches,  practices  and 
technology  used  to  identify  failure  modes, 
detect  item  degradation  modes  and  measure 
rates  of  degradation.  The  selection  of  items 
for  the  aging  tests  should  be  based  on  estab¬ 
lished  criteria:  e.g.,  item  criticality,  known 
storage,  handling  or  operational  limitations, 
unknown  long  term  performance  (stability)  of 
an  item,  and  intended  operation,  storage  or 
handling  in  uncontrolled  or  undesirable  en¬ 
vironments. 

The  Service  Life  Evaluation  Test  Program 
is  a  form  of  aging  test  program ;  it  is  a  coor¬ 
dinated  hardware  fleet  return  program,  to 
provide  early  detection  of  item  deterioration. 

3. 6. 2.6  SPALT  Test  Requirements 

The  Strategic  System  Projects  Alteration 
(SPALT)  Program  governs  the  necessary  tasks 
for  configuration  control  and  status  account¬ 
ing  of  configuration  changes  to  SWS  hardware 
and  software.  The  SPALT  program  is  a  con¬ 
trolled  system  requiring  testing  of  configura¬ 
tion  changes;  it  is  defined  by  and  required  to 
be  in  accordance  with  SSPINST  P4720. 1  and 
SSP1NST  41 30.7. 
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3.6.3  Pre-Deployment  Phase  Tests 

Test  programs  during  the  pre-deployment 
phase  provide  an  assessment  of  the  Strategic 
Weapon  System  as  it  is  installed  in  the  sub¬ 
marine. 

3.6.3. 1  Shipyard  Installation  Or  Overhaul 
Test  Program 

As  the  SWS  is  installed  aboard  each  sub¬ 
marine  during  new  construction,  conversion 
or  overhaul,  the  SWS  undergoes  pre-deploy¬ 
ment  testing  in  accordance  with  the  Shipyard 
Installation  Test  Program  (SITP)  or  Shipyard 
Overhaul  Test  Program  (SOTP).  The  objective 
of  these  tests  is  to  demonstrate  weapon  sys¬ 
tem  operability  and  performance,  in  as  nearly 
a  tactical  closed-loop  configuration  as  possi¬ 
ble,  at  dockside  and  at  sea.  This  program  tests 
the  SWS,  including  ship  support  systems,  with 
the  aid  of  an  Active  Inert  Missile.  The  respon¬ 
sibilities  of  all  activities  concerned  with  SITP/ 
SOTP,  as  well  as  the  specific  test  require¬ 
ments,  are  delineated  in  an  Integrated  Test 
Program  Outline. 

SITP/SOTP  is  divided  into  six  phases;  in¬ 
spection,  static,  grooming,  subsystem  opera¬ 
tion,  system  interface  and  system  operation 
tests.  Inspection  includes  visual  checks  of  all 
subsystems  for  damage,  arrangement,  orien¬ 
tation,  clearances,  accessibilities,  installation 
in  accordance  with  applicable  drawings,  and 
tests  of  cable  continuity.  Trident  Measure¬ 
ment  Program  tests,  to  verify  that  critical 
coordinated  features/parameters  (e.g.,  access 
door  alignment,  support  group  height)  have 
been  fulfilled,  are  also  included.  Static  tests 
are  integrity  tests  for  submarine  electrical 
power  supplies  and  piping  systems;  they  in¬ 
clude  energizing  electrical  power  supply  cir¬ 
cuits  to  check  for  isolation  to  ground,  proper 
voltage,  frequency  and  phase  rotation,  and 
piping  systems  to  check  for  strength.  Groom¬ 
ing  tests  insure  satisfactory  operation  of  all 
equipment  and  circuits  within  a  subsystem, 
including  controls,  displays,  sequencing, 
alarms  and  instrumentation.  Subsystem  opera¬ 
tion  tests  demonstrate  that  the  subsystems 
perform  within  specified  tolerances  in  normal 
casualty  and  test  modes.  System  interface 
tests  demonstrate  that  proper  mechanical. 


electrical,  fluid  and  optical  interconnection 
and  transmission  exists  between  subsystems. 
System  operation  tests  demonstrate  system 
operability  and  performance. 

3. 6. 3. 2  Demonstration  And  Shakedown 
Operation  Tests 

Demonstration  and  Shakedown  Operation 
(DASO)  Tests,  consisting  of  test  missile 
launches,  are  intended  to  demonstrate  the 
readiness  of  the  complete  weapon  system  for 
deployment.  They  evaluate  the  complete 
weapon  system,  including  crew  performance, 
all  system  interfaces,  shore-based  support 
caoability  and  missile  accuracy  and  reliability 
under  operational  conditions.  DASO  tests  are 
performed  after  initial  installation  of  the 
strategic  weapon  system  in  a  newly  con¬ 
structed  submarine  and  after  each  submarine 
overhaul  cycle  or  conversion,  to  verify  readi¬ 
ness  of  production  systems  and  crews. 

DASO  is  the  earliest  of  the  fleet  test  pro¬ 
grams  employed  by  SSPO  to  evaluate  the  SWS 
in  service.  DASO  tests  provide  baseline  per¬ 
formance  data  in  operational  environments. 
They  also  provide  crew  training  and  aid  the 
development  and  evaluation  of  tactical  oper¬ 
ating  procedures  and  related  documentation. 
They  serve  to  certify  both  the  weapon  system 
and  crews  for  deployment. 

DASO  demonstrates  the  safety  and  service¬ 
ability  of  the  weapon  system,  tests  the  ade¬ 
quacy  of  modifications  and  improvements, 
and  enables  crews  to  determine  and  demon¬ 
strate  weapon  system  performance  in  new 
missions  or  new  modes  of  operation.  DASO 
operations  furnish  preliminary  missile  flight 
reliability  and  accuracy  data  to  support 
recommendations  for  Single  Integrated  Opera¬ 
tions  Plan  (SlOP)  planning  factors.  DASO 
tests  are  specially  instrumented  and  moni¬ 
tored  and  are  designed  to  be  as  nearly  repre¬ 
sentative  of  tactical  situations  as  possible. 

DASO  (TRIDF.NT  11)  consists  of  seven 
phases  of  test  operations: 

1)  Loadout  loading  and  alignment,  mis¬ 
sile  acceptance  tests  and  inspections. 

2)  Navigation  Operations  (NAVOPS) 
testing  and  calibration  of  the  navigation  sub¬ 
system. 

3)  In-tube  Conversion  (IT(')  convert  to 
test -missile  configuration. 
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4)  Preparation  and  Readiness  1  valuations 
(PREPs)  installation,  checkout  and  accep¬ 
tance  tests  of  instrumentation. 

5)  Tactical  F*crcise  demonstrate:  (a) 
readiness  of  equipment,  crew  and  documen¬ 
tation,  (b)  interface  and  performance  capa¬ 
bility,  (c)  countdown  procedures. 

6)  Launcher  Operations  -  test  Firings  with 
impact  location. 

7)  Post-DASO  Operations  -  restore  SWS 
to  tactical  configuration. 

The  navigation  subsystem  is  evaluated  dur¬ 
ing  DASO  relative  to  routine  upkeep  and 
patrol  operations.  This  includes  the  crew’s 
ability  to  conduct  standard  monitoring  and 
calibration,  and  the  subsystem’s  ability  to 
support  tactical  mission  accuracy  and  re¬ 
liability  objectives.  Estimates  are  made  of 
navigation  functional  errors  and  uncertainties 
at  launch,  and  the  adequacy  and  accuracy  of 
applicable  documentation  is  evaluated. 

The  fire  control  subsystem  is  self-tested 
using  the  keyboard  and  test  operating  panel. 
The  subsystem’s  contribution  to  dynamic 
errors  (position,  velocity,  heading)  are  esti¬ 
mated,  as  is  the  reliability  of  the  subsystem 
and  its  constituent  equipments. 

A  variety  of  missile  evaluations  are  made, 
including  guidance  alignment  and  readiness 
tests.  Performance  of  the  guidance  servo-loop, 
attitude  and  acceleration  control,  computer, 
instrumentation  and  other  guidance  compo¬ 
nents  is  measured. 

Overall  functional  performance  of  the  mis¬ 
sile  and  guidance  subsystems  in  flight  is  de¬ 
termined  by  missile,  submarine  and  support 
ship  instrumentation,  and  by  range  instrumen¬ 
tation,  which  continuously  measures  and 
records  missile  performance  and  down-range, 
cross-range  and  height-of-burst  miss  distance?. 

The  launcher  subsystem  is  evaluated  foi 
performance  of  missile  gas  system,  ejection, 
in-tube  and  underwater  flight,  closure  detona¬ 
tion  and  event  timing.  The  ship’s  hovering  and 
compensation  functions  are  also  evaluated. 

Sonar  evaluation,  communications  readi¬ 
ness  exercises  and  special  tests  for  hardware 
proofing  and  other  purposes  may  also  be  con¬ 
ducted  in  conjunction  with  DASO  operations. 

All  hardware  and  software  failures,  repair 
actions,  and  anomalous  conditions  are  tabu¬ 
lated  during  DASO.  Statistical  measures  of 
reliability,  accuracy  and  performance  are 


calculated  for  comparison  with  expected 
performance.  Design  defects  or  deficiencies, 
performance  abnormalities,  documentation 
deficiencies  and  similar  problems  are  identi¬ 
fied  and  corrective  action  recommendations 
are  made.  SPALTS/TVAs  are  evaluated  to 
verify  that  they  have  no  adverse  effects  on 
performance. 

Upon  SSPO  direction,  contractor  personnel 
may  be  requested  to  participate  in  DASO 
tests,  to  provide  information  concerning  oper¬ 
ation  and  status  of  equipment  furnished  by 
their  company,  and  advice  and  support  on 
problems  that  may  arise  during  DASO. 

3.6.4  Deployment  Phase  Tests 

Fleet  test  programs  during  the  deployment 
phase  provide  a  continuing  assessment  of  the 
functional  status  of  items  in  the  logistic  flow 
and  on  station,  and  the  effects  of  long  term 
operational  environments  on  items  at  various 
logistic  locations. 

These  tests  support  the  fleet  inventory  and 
deployment  program.  They  include  field  test¬ 
ing  of  production  items  in  the  logistic  flow. 
The  tests  range  from  comprehensive  system 
tests  at  shore-based  facilities  to  limited  system 
tests  aboard  submarines.  Their  primary  objec¬ 
tive  is  to  assure  satisfactory  functioning  of 
tactical  systems  throughout  the  operational 
cycle.  Tests  are  conducted  at  maintenance 
levels  1 , 2,  and  3. 

Level  1  tests  (Patrol  Tests)  conducted 
aboard  the  submarine  verify  operational  readi¬ 
ness  of  the  system.  Level  2  tests  (Recertifica¬ 
tion  Tests),  at  the  refit  facility  and  at  the  ten¬ 
der,  assure  continued  operational  readiness  of 
the  complete  weapon  system.  These  tests,  con¬ 
ducted  during  selected  refit  periods,  are  simi¬ 
lar  to  Level  1  tests.  A  capability  exists  on 
tenders  to  perform  package  surveillance, 
prior-to-issue  tests  and  package  failure  verifi¬ 
cations.  Level  3  tests  conducted  at  shore- 
based  facilities  include  non-destructive  testing 
of  ordnance  and  propulsion  items.  They  vali¬ 
date  and  certify  factory  produced  items  for 
deployment  and  inventory. 

Tests  are  conducted  at  all  three  mainte¬ 
nance  levels  with  support  equipment  that  has 
been  certified  for  use  in  the  production  test 
program.  Deployment  phase  tests  are  con¬ 
ducted  by  government  and  contractor  person- 
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nel.  Testing  criteria  and  techniques  are  based 
on  information  developed  by  the  integrated 
test  program  during  development  and  produc¬ 
tion  phases.  Test  results  are  transmitted  to 
the  contractor  for  use  in  tracking  product 
quality  trends. 

Tests  during  patrols  include  routine  mainte¬ 
nance,  operability  and  checkout  tests  and 
Weapon  System  Readiness  Tests  (WSRT).  Data 
from  patrol  tests  are  integrated  with  DASO 
and  Operational  Test/Follow-On  Operational 
Test  (OT/FOT)  data  to  assess  the  accuracy 
and  reliability  of  the  deployed  fleet.  After  the 
patrol,  the  SSBN  crew  prepares  a  data  pack¬ 
age,  which  includes  handwritten  logs.  Fire 
Control  and  Navigation  hardcopy  printouts, 
failure  and  problem  reports,  data  and  perfor¬ 
mance  tapes  (audio  and  digital).  All  equip¬ 
ment  downtimes  and  repair  times  during 
patrol  are  recorded,  system  and  subsystem 
readiness  indices  are  calculated.  “Quick  look” 
evaluations  are  prepared  for  each  subsystem. 

Patrol  testing  analysis  has  several  objec¬ 
tives: 

1)  To  provide  information  based  on  per 
formance  of  the  SWS  under  actual  patiol 
conditions,  to  derive  SIOP  planning  factors. 

2)  To  provide  operational  commanders 
with  evaluations  of  individual  SSBNs  and 
classes  of  SSBNs  on  patrol. 

3)  To  provide  crews  with  analyses  of  their 
ship’s  performance,  to  enable  them  to  im¬ 
prove  operating  and  maintenance  procedures 
and  to  compare  their  ship’s  performance  with 
others  of  the  same  class. 

4)  To  provide  the  subsystem  branches  at 
SSPO  with  analyses  of  the  performance  of 
their  subsystems  and  constituent  equipments 
in  the  tactical  environment. 

5)  To  supplement  crew  reports  relative  to 
equipment  problem  areas. 

Reports  and  other  source  data  from  patrol 
performance  analyses  may  be  made  available 
to  contractors  for  independent  evaluation  of 
their  equipment. 

3.6.5  Special  Operations  Phase  Tests 

Periodically,  submarines  are  ordered  off 
patrol  to  perform  an  Operational  Test  (OT)  or 
Follow-On  Operational  Test  (FOT). 
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Operational  tests  and.  at  reduced  Irc- 
quency.  Follow-On  Tests  are  performed  to: 

1)  Obtain  reliability,  performance,  and 
accuracy  data  and  SIOP  planning  factors  lor 
the  weapon  system  and  its  constituent  sub¬ 
systems  under  representative  tactical  condi¬ 
tions. 

2)  Verify  that  reliability,  performance, 
accuracy  and  planning  factors  do  not  signifi¬ 
cantly  degrade  during  the  life  of  the  weapon 
system. 

3)  Determine  the  adequacy  of  tactical  soft¬ 
ware,  procedures  and  technical  support  docu¬ 
mentation. 

Operational  tests  involve  launches  of  test 
missiles  into  areas  monitored  by  reentry  im¬ 
pact  location  systems.  The  tests  are  con¬ 
ducted  in  as  operationally  realistic  a  manner 
as  pcssible,  with  nontactical  operations  held 
to  a  minimum.  Testing  is  structured  to  span 
the  range  of  mission  variables  (e.g..  range,  loft 
angle,  reentry  velocity).  Unplanned  deviations 
are  evaluated  on  a  case  by  case  basis,  so  that 
statistical  “outliers”  can  be  treated  properly. 
For  example,  data  from  tests  representing 
unreliable  performance  or  other  deviations 
from  normal  performance  is  usually  excluded 
from  accuracy  calculations.  The  effects  of 
system  modifications  are  evaluated,  and  esti¬ 
mates  of  the  weapon  system’s  performance 
factors  and  reliability  are  adjusted  accord¬ 
ingly. 

The  accuracy  with  which  the  reentry  body 
is  delivered  to  the  target  area  is  evaluated. 
Accuracy  data  are  used  to  establish  sufficient 
impact  and  burst-height  error  statistics  to 
make  valid  estimates  of  accuracy  under  ex¬ 
pected  tactical  conditions,  and  to  determine, 
to  the  extent  permitted  by  available  instru¬ 
mentation,  error  sources  within  the  system 
on  a  subsystem  and  flight  phase  basis. 

Operational  tests  (OT)  are  used  to  establish 
performance  factors  based  on  a  statistically 
significant  sample  of  representative  tactical 
missiles  for  use  by  the  Joint  (duels  ot  Stall 
(JCS)  in  the  SIOP.  Follow -on  operational 
tests  (FOT)  are  conducted  to  vettly  that 
system  performance  has  not  ,  bungee  with 
service  life. 
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Section  4 

TEST  PLANNING  BY  LINE  ORGANIZATIONS 


This  section  sets  forth  guidance  for  line 
organizations  to  support  an  Integrated  Test 
Program.  Techniques,  considerations  and  pre¬ 
cautions  that  should  be  utilized  or  observed 
when  planning  tests  are  presented  and  dis¬ 
cussed. 

4.1  GENERAL  CONSIDERATIONS 

The  requirements,  guidelines  and  instruc¬ 
tions  discussed  previously  to  manage  and 
control  the  test  program  must  be  made  avail¬ 
able  to  affected  line  groups.  Training  sessions 
should  be  held  to  familiarize  line  personnel 
with  the  intent,  structure  and  operation  of 
the  ITP.  Attention  should  be  directed  toward 
defining  the  types  of  tests  as  cited  in  Sec¬ 
tion  3,  completing  information  forms,  pre¬ 
paring  test  plans,  procedures  and  reports, 
maintaining  the  integrity  of  the  tests,  identi¬ 
fying  test  requirements,  consolidating  tests 
where  possible,  and  using  standardized  data 
recording  forms. 

4.2  TEST  DOCUMENTATION 

Test  documents,  such  as  information 
forms,  plans  and  procedures,  are  the  means 
for  formally  disseminating  test  needs,  and  for 
planning,  performing  and  reporting  tests. 
These  documents  fall  into  two  categories: 
test  documents  used  as  tools  by  the  ITPB  and 
documents  required  by  contract. 

4.2.1  Documents  Used  By  The  ITPB 

To  plan  and  administer  an  integrated  test 
program,  the  ITPB  requires  test  information 
to  be  supplied  by  the  line  organizations.  The 
timeliness  of  that  information  is  extremely 
important.  The  following  paragraphs  discuss 


documents  line  organizations  can  provide  to 
aid  the  ITPB  and  the  operation  of  the  ITP. 

4. 2.1.1  Test  Planning  Information  Form 

Test  planning  information  is  completed  and 
submitted  to  the  ITPB,  via  the  test  facilities 
manager,  as  soon  as  possible  after  the  need  for 
a  test  has  been  identified.  Figure  4-1  is  an 
example  of  an  information  form.  It  includes 
areas  for  evaluation  by  .the  test  facilities 
manager  and  the  ITPB.  Areas  of  concern  or 
disagreement  should  be  resolved  as  soon  as 
possible.  Among  information  items  that 
should  be  provided  are: 

Independent  Test  Facilities  -  Independent 
test  facilities  provide  alternatives  or  supple¬ 
ments  to  in-house  testing;  they  may  provide 
certain  advantages  over  in-house  facilities: 

a.  Relief  of  high  demand  on  in-house  facil¬ 
ities. 

b.  Experienced  testing  personnel  having 
technical  expertise  that  may  not  be  available 
in-house. 

c.  Existing  test  plans  and  procedures  that 
may  be  reused  or  modified. 

d.  High-cost  sophisticated  test  systems, 
which  can  reduce  test  labor  and  automate 
production  of  required  verification  documen¬ 
tation. 

e.  Greater  flexibility  in  implementing  an 
ITP,  by  contractors  with  limited  testing  capa¬ 
bilities. 

In  analyzing  the  need  for  and  constraints 
on  the  use  of  independent  test  facilities,  the 
ITPB  should  evaluate  the  capability  and  repu¬ 
tation  of  the  facility  proposed.  Areas  of 
concern  are  schedule  availability,  logistic  com¬ 
plexities  and  costs,  required  on-site  test  sur¬ 
veillance  by  witnessing  personnel,  metrology 
capabilities,  expertise  in  the  area  of  the 
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FROM:  Item  Designer 
TO:  ITPB 

VIA:  Test  Facilities  Manager 


TEST  PLANNING  INFORMATION  FORM 


Item  No. _ Form  No - 

Item  Name _ 

No.  of  Items  Required  Per  Supplier _ 

□  Item  on  hand  at  present  time. 

□  Item  on  order,  expected  receipt  date _ 

□  Item  produced  in-house,  expected  receipt  date 
Location  of  Test  Facility 

CD  In-house  __________________ 

CD  Independent  Facility _ 

Test  Number _ 

Test  Name _ 

Test  is  for  corrective  action  verification? 

Test  Objective _ 


_  Length  of  Test 

No.  of  Suppliers _ 


CD  Source _ 

.  Availability  Date  . 
Test  Type  _ 


CD  Yes 


.  Total  Cost  of  Test . 


.Contracted?  CD  Yes  CD  No 


_  CD  Nondestructive 
Te*n*  □  Destructive 


Ability  of  design  to  pass  test. 


Satisfactory? 


CD  Low 


CD  Medium 


CD  High 


ITPB  Evaluation 


Environmental  Requirements 


CD  Humidity _ 

□  Shock  _ 

CD  Acceleration _ 

CD  Vibration _ 

CD  Temperature 

□  Altitude 

CD  User  ( For  Software) _ 

□  Other _ 


Test  Plan  Available:  Preliminary _ Date _ 

Test  Procedure  Available:  Preliminary  _ Date _ 

Pre-test  conditioning  required?  CD  No  CD  Yes  Describe: 


GFM  Required?  CD  No  CD  Yes 

GFM  Description _ 

GFM  Present  Action _ 


Requested?  CD  Yes  CD  No 


Date  needed? . 


Can  obtain  Reliability  Evaluation  Data  from  test?  CD  Yes  CD  No 
obtain  Reliability  Data? _ 


If  No,  what  test  changes  are  necessary  to 


Can  obtain  Maintainability  Evaluation  Data  from  test?  CD  Yes  □  No  If  No,  what  test  changes  are  necessary  to 

obtain  Maintainability  Data? _ 


Can  obtain  Accuracy  Data  from  tests?  CD  Yes  CD  No 
Accuracy  Data? _ 


If  No,  what  test  changes  are  necessary  to  obtain 


Can  data  for  the  evaluation  of  the  maintenance  package  be  obtained  from  test?  CD  Yes  CD  No 

If  No,  what  changes  are  necessary  to  obtain  data? _ 

Disposition  of  test  items  at  completion  of  test? _ 


Traceability  of  Test  Requirement 
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required  type  of  test,  and  the  ability  to  test  able.  Status  updates  can  be  provided  by  later 

at  the  necessary  assembly  level.  revisions  of  the  planning  form. 

Environmental  Requirements  -  The  line 

organization  provides  the  planned  environ-  4.2. 1.2  Delinquent  Test  Information  Notice 
mental  and  test  limits  for  each  specific  en¬ 
vironment.  The  1TPB  must  be  completely  When  it  is  difficult  to  get  the  required  test 

familiar  with  the  program  requirements  to  inputs  from  a  line  group,  the  ITPB  should 

assure  that  the  planned  limits  are  correct.  Any  issue  a  delinquent  test  information  notice  to 

discrepancies  should  be  resolved  immediately.  encourage  the  submission  of  test  information. 

Government  Furnished  Material  ( GFM )  -  Efforts  should  be  made,  however,  to  resolve 

Government  furnished  material  is  usually  re-  delinquencies  by  direct  contacts  whenever 

quired  early  by  the  contractor,  and  made  possible.  Once  the  notice  is  issued,  a  response 

available  as  part  of  the  contract.  This  area  of  period  of  not  more  than  two  weeks  should  be 
the  planning  information  form  is  used  to  allowed.  If  the  delinquency  is  not  resolved 

verify  that  needed  GFM  has  been  placed  within  that  time  period,  action  should  be  in- 

under  contract.  If  this  is  not  the  case,  steps  itiated  at  the  program  manager  level, 

should  be  taken  to  modify  the  contract  or  4.2.1.3  Test  Program  Change  Proposal 

ITP. 

Reliability,  Maintainability  Or  Accuracy  A  Test  Program  Change  Proposal,  such  as 

Assessment  Data  -  The  line  organization,  Figure  4-2,  can  be  initiated  by  anyone  in  the 

working  with  the  cognizant  specialist  groups,  program.  The  intent  of  a  proposal  may  be  to 

should  review  the  intended  test  for  usability  reduce  test  cost,  facility  burden,  manpower 

of  results  to  assess  reliability,  maintainability  requirements,  or  time  constraints.  A  proposal 

or  accuracy.  The  ITPB  may  recommend  should  relate  to  one  or  more  specific  tests;  it 

changes  to  the  proposed  test  to  make  the  test  should  show  how  the  present  test  position  can 

results  more  useful  for  assessment  of  reli-  be  improved  by,  for  example,  combining  tests 

ability,  maintainability  or  accuracy,  diagnos-  to  achieve  the  required  test  program  objec- 

tic  programs  or  documentation.  tives.  Supporting  data  to  show  that  previous 

Traceability  Of  Test  Requirement  -  The  requirements  will  be  met  by  the  new  proposal 

line  organization  should  provide  enough  detail  must  be  included  on  the  form.  Each  respon- 

in  this  section  so  that  the  test  requirement  sible  evaluation  activity  (e.g.,  item  designer, 

can  be  traced.  More  columns  may  be  neces-  ITPB,  test  facilities  manager)  reviews  the 

sary  to  reflect  subtier  specifications  and  para-  proposal.  The  approved  proposal  is  then  di¬ 
graphs.  If  the  planned  test  is  an  extensive  tributed. 

verification  of  item  requirements,  a  separate 

attachment  may  be  added  to  provide  the  4.2.2  Test  Documentation  Required  By 
traceability  of  each  requirement  (parameter  Contract/CDRL 

or  function)  to  be  verified. 

Evaluations  -  Evaluations  of  planned  tests  Contract  data  requirements  are  specified  in 

by  the  ITPB  and  by  test  facility  personnel  are  NAVSEA  OD21549A  and  contracted  for  by 

most  critical  early  in  a  program.  Planning  inclusion  on  the  Contract  Data  Requirements 

errors  may  be  difficult  to  correct  due  to  the  List  (CDRL).  Figure  2-1  contains  a  listing  of 

length  of  time  required  to  contract  for  the  NAVSEA  OD21549A  required  docu- 

services,  purchase  capital  equipment  or  rerun  ments.  Contractual  test  documentation  in¬ 
tests.  The  ITPB  chairman  should  assure  that  eludes  test  plans,  procedures  and  reports, 

all  organizations  affected  are  aware  of  evalua-  These  documents  are  generally  prepared  by 

tions  made  by  the  board  and  of  actions  being  line  organizations, 

taken  to  resolve  differences. 

Planning  information  provided  to  the  ITPB  4.2.2. 1  Test  Plans 

should  be  as  accurate  as  possible,  but  submis¬ 
sion  of  planning  forms  should  not  be  delayed  A  copy  of  each  test  plan  is  submitted  to 

because  certain  information  is  not  yet  avail-  the  ITPB  for  approval.  Test  plans  are  needed 


V  * 
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Figure  4-2.  Example  Of  A  Test  Program  Change  Proposal 
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TEST  PLAN 

The  test  plan  shall  identify  and  describe  planned  contractor  activities  for  implementa¬ 
tion  of  test  requirements.  As  a  minimum,  the  test  plan  should  include  the  following: 

a.  Type  of  test;  e.g.,  Hardware-EET,  Software-Validation. 

b.  Applicable  documents. 

*c.  Test  objectives  and  requirements  of  each  test,  to  include  why  the  test  is 
necessary,  what  the  test  will  achieve,  traceability  of  requirement. 

*d.  Test  item  configuration  description  and  number  of  items  to  be  tested  (if 
more  than  one  supplier,  identify  all  suppliers  and  quantity  from  each  which 
will  be  tested). 

e.  Limited  life  item  identification. 

f.  Conditions  to  be  met  before  testing  begins,  e.g.,  item  design  maturity,  pre¬ 
test  conditioning,  approvals,  etc. 

#g.  Test  conditions,  environmental,  operational  (c’>finition  of  each  operating 
mode)  and  performance  profiles,  and  duty  cycle. 

h.  Order  in  which  elements  of  the  test  will  be  performed. 

*i.  Test  conduct  ground  rules,  failure  criteria,  analysis  techniques  for  interface 
boundaries  (between  test  set-up  and  item  under  test)  for  assignment  of 
failure. 

*j.  Parameters  to  be  measured  and  required  accuracy. 

*k.  Data  collection,  analysis  and  corrective  action  system  to  be  used,  including 
failure  occurrence  control. 

*1.  Test  duration. 

m.  Criteria  for  continuing  test  in  the  event  a  failure  or  stoppage  occurs. 

n.  Samples  of  data  collection  forms. 

o.  Suitability  of  the  test  for  reliability,  maintainability  and/or  accuracy  assess¬ 
ment. 

p.  Any  planned  maintenance  to  test  item(s)  or  test  facility  during  test. 

*q.  Test  facility  and  equipment  descriptions  and  requirements,  including  safety 
monitoring  for  electrical,  mechanical  and  environmental  interfaces. 

r.  Government  Furnished  Material  requirements. 

*s.  Test  schedules  and  milestones. 

t.  Disposition  of  tested  items  upon  completion  of  test. 

u.  Contractor  and  procuring  activity  participation: 

(1)  specific  responsibilities, 

(2)  access  to  test  area  and  item  under  test, 

(3)  training  or  familiarity  requirements. 

v.  List  of  test  procedures  required. 

w.  List  of  test  reports  to  be  issued. 

•Required  By  NAVSEA  OD21549A 

Figure  4-3.  Typical  Content  Of  A  Test  Plan 
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for  each  item  of  hardware  and  software  that 
is  to  be  tested.  Planning  should  be  performed 
in  sufficient  detail  to  enable  the  ITPB  to 
evaluate  the  adequacy  of  each  test,  and  to 
determine  whether  and  how  individual  plans 
can  be  modified  for  an  optimum  overall  pro¬ 
gram.  Test  plans  for  items  that  are  to  receive 
similar  testing  may  be  consolidated  to  the 
extent  that  information  common  to  them  is 
not  repeated,  but  is  cited  by  reference.  Figure 
4-3  lists  the  content  requirements  of  a  test 
plan. 

4. 2.2.2  Test  Procedures 

A  test  procedure  should  be  prepared  for 
each  test,  to  serve  as  a  step-by-step  guide  for 
the  test  personnel  who  will  perform  the  test 
and  to  enable  others  to  monitor  conduct  of 
the  test.  Test  procedures  should  be  tailored  to 
the  technical  level  of  the  personnel  who  will 
perform  the  test.  Thus,  style  and  degree  of 
detail  may  range  from  “insert  probe  A  at 
point  B”  to  “run  gain  and  insertion  loss 
checks  using  a  model  4200  RF  microwatt- 
meter”.  When  there  is  a  question  as  to  the 
training  level  of  the  test  personnel  who  will 
perform  a  test,  it  is  better  to  error  on  the  side 
of  too  much  detail  in  the  test  procedure. 

An  adequate  procedure  enables  a  properly 
trained  user  to  conduct  the  test  correctly 
without  reference  to  other  documents.  Neces¬ 
sary  test  equipment  and  instruments  are  speci¬ 
fied,  the  test  set-up  is  shown  schematically,  all 
required  measurements  and  tolerances  are 
listed.  The  data  sheets  to  be  used  to  record 
item  performance  in  the  test  are  included  as 
part  of  the  procedure.  Figure  4-4  lists  typical 
items  of  information  that  should  be  contained 
in  a  test  procedure. 

4. 2. 2.3  Test  Reports 

A  test  report  is  a  formal  document  report¬ 
ing  and  interpreting  the  performance  of  one 
or  more  tests.  The  ITPB  will  usually  be  asked 
to  confer  approval  on  test  results,  or  to  direct 
corrective  action  and  retesting,  without  wait¬ 
ing  for  a  test  report  to  be  produced.  Review 
of  the  test  procedure,  test  datasheets,  failure 
reports,  failure  analysis  reports,  memos  or 
other  documents  evidencing  compliance  with 


required  corrective  actions,  generally  com¬ 
prises  a  sound  basis  for  timely  ITPB  action. 
The  ITPB  should  require  however,  that  all 
tests  be  documented  by  means  of  test  reports 
within  a  few  weeks  of  completion.  A  copy  of 
each  report  is  filed  by  the  ITPB  as  part  of  the 
item  test  data  package.  Tests  stretching  over 
several  months  or  more  should  be  reported  at 
intervals,  to  keep  the  ITPB  and  management 
aware  of  status  and  progress  against  program 
milestones. 

Figure  4-5  lists  the  typical  information  con¬ 
tent  of  a  test  report. 

4.3  PLANNING  SOFTWARE  TESTS 

4.3.1  Software  Module  Test  Plans  And 
Procedures 

The  content  of  software  module  test  plans 
and  procedures  is  basically  similar  to  that 
indicated  for  hardware  tests  in  Figure  4-3  and 
4-4,  modified  to  meet  the  special  needs  of 
software  tests.  Modifications  include: 

a.  When  identifying  support  facilities  and 
equipment,  include  support  software 
requirements  such  as  simulation,  data 
recording,  data  reduction,  data  analysis 
programs,  known  test  cases  or  test  case 
libraries. 

b.  All  outputs,  at  each  evaluation  point, 
should  be  specified  with  tolerances  for 
the  defined  inputs.  Outputs  should  be 
verified  for  limits,  accuracies  and  timing. 

c.  Meaningfulness  of  outputs,  ease  of  in¬ 
terpretation,  failure/error  diagnostics, 
consistency  and  uniformity  of  syntax, 
conventions,  semantics,  format,  style 
and  abbreviations. 

d.  Acknowledgement  of  all  human  inputs 
at  the  user  interface. 

e.  Verification  of  the  software  security  pro¬ 
visions. 

f.  Test  inputs  should  include  both  legal  and 
illegal  inputs  and  should  be  provided  at 
a  rate  able  to  stress  the  module(s)  under 


4-7 


NAVSEA  OD  42282A 


TEST  PROCEDURE 

Test  procedures  provide  detailed  technical  directions  for  implementing  the  required 
test.  Procedures  contain  step-by-step  instructions  of  how  the  test  will  be  set-up,  initi¬ 
ated  and  performed.  The  test  procedure  should,  as  a  minimum,  include  the  following: 

a.  Applicable  documents,  including  specifications,  instructions,  procedures, 
handbooks,  manuals,  military  specifications,  software  program  listings,  etc. 

b.  A  brief  description  of  all  units  comprising  the  item(s)  under  test  and  a 
specific  listing  of  those  units/items  which  will  be  placed  on  test  and  the 
up-to-date  configuration  (drawing  list  including  approved  changes,  waivers 
and  deviations). 

c.  Test  and  monitoring  equipment  to  be  used,  including  manufacturer,  model 
number,  serial  number  and  calibration  requirements. 

*d.  Interconnecting  cable  diagrams  of  complete  test  set-up  including  item 
under  test  and  test  monitoring  equipment. 

*e.  Special  equipment  or  facilities  required,  such  as  fixtures,  etc. 

f.  Any  computer  software  used  in  the  test. 

g.  List  of  any  Government  Furnished  Materials  to  be  used  during  test. 

h.  Assumptions  concerning  test  or  deviations  from  specifications. 

i.  Pre-test  conditioning. 

#j.  Electrical,  mechanical  and  environmental  stress  levels. 

*k.  Normal  checkout  procedures  for  item(s)  under  test. 

*1.  Verification  of  test  set-up,  including  power-up  sequence. 

m.  Allowable  adjustments  during  test. 

n.  Preventive  maintenance  measures  to  be  performed,  if  allowable  by  specifi¬ 
cation,  during  test. 

*o.  Performance  parameters  to  be  measured,  accuracy,  frequency  of  measure¬ 
ment,  and  method.  Where  visual  observation  is  to  be  used,  sufficient  criteria 
for  evaluation  must  be  provided. 

*p.  Data  to  be  recorded  during  tests  and  samples  of  reports  or  log  forms  to  be 
used;  e.g.,  Test  Log  and  Data  Record,  Failure  Record,  Failure  Tag,  Failure 
Report. 

*q.  Performance  parameter  limits  beyond  which  a  failure  has  occurred. 

*r.  Pass/fail  criteria. 

s.  Action  to  be  taken  in  event  of  failure  (or  software  error). 

t.  Action  to  be  taken  if  a  reject  decision  is  reached,  including  corrective 
action  plan  and  retest  provisions. 

*u.  Action  to  be  taken  in  the  event  of  test  interruption. 

*v.  Whether  testing  will  be  continuous  or  interrupted  by  work  shifts. 

*w.  Applicable  safety  precautions  for  personnel  and  facility  protection. 

*  Required  by  NAVSEA  OD21549A. 


Figure  4-4.  Typical  Content  Of  A  Test  Procedure 
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I 


TEST  REPORT 

The  test  report  is  the  formal  record  of  the  results  of  an  item(s)  test.  The  test  should, 
as  a  minimum,  include  the  following: 

*a.  A  reference  to  the  applicable  test  plan  and  procedures. 

b.  Test  article  identification  and  full  description  of  test  specimens  utilized, 
including  any  deviations  from  the  configurations  specified  in  the  appli 
cable  test  plan. 

c.  Date  and  geographical  location  of  test. 

d.  Statement  of  test  objectives;  including  type,  unit  of  measure,  and  quantita 
tive  goals/requirements. 

e.  Discussion  of  methods  and  conditions  of  the  test,  including  environmental 
levels,  test  profile,  methods  of  evaluating  the  data  obtained  and  comparison 
of  the  conditions  with  those  anticipated  in  ultimate  deployment  and  use  of 
the  contract  item. 

f.  Results  obtained,  including  identification  and  discussion  of  objectives 
demonstrated  satisfactorily  and  those  not  demonstrated  satisfactorily. 

*g.  Identification  of  significant  events,  problems  and  any  departures  from  the 
test  procedure. 

h.  Pre-conditioning  results. 

i.  Corrective  action  planned  or  taken. 

*j.  Results  of  data  analysis,  failure  diagnosis,  conclusions  and  recommenda 
tions. 

k.  Requirement  for  retest. 

l.  Results  of  retest. 

m.  Copy  of  all  test  data,  e.g.,  test  logs  and  data  records. 

*n.  Copies  of  waivers,  deviations  and  failure  reports  pertaining  to  the  test. 

•Required  by  NAVSEA  OD21549A. 

Figure  4-5.  Typical  Content  Of  A  Test  Report 

test  at  or  above  their  design  limits.  Real  (1)  Any  extensive  manipulation  of  con- 

data  is  preferable  to  synthetic  data.  trols  and  settings. 


g.  Incorporation  of  error  corrections  and 
verification  of  error  corrections  should 
remain  within  the  control  of  the  con¬ 
figuration  management  system,  to  re¬ 
duce  the  probability  of  the  error  correc¬ 
tion  process  introducing  new  errors  into 
the  software. 

h.  Step-by-step  instructions  should  address 
the  man-machine  interface  and  include: 


(2)  Each  step  of  testing,  listed  as  an  indi¬ 
vidual  instruction. 

(3)  Evaluation  criteria  for  any  step 
where  evaluation  is  required 

(4)  The  evaluation  technique,  such  as 
templates  for  use  on  hard  copy 
outputs. 


NAVSEA  OD  42282A 


i.  Failure/enror  determination,  to  generally 
include  any  stoppage  of  the  test. 

j.  Identification  of  government  furnished 
material  including  support  software 
(tools,  service  aids,  etc.). 

k.  Testing  should  provide  data  to  analyze 
the  error  rate  of  the  software,  classify 
errors,  and  measure  the  rate  of  reliability 
growth  (error  rate  reduction). 

4.3.2  Software  System  Test  Plans  And 
Procedures 

Software  system  test  plans  and  procedures 
should  include  content  as  indicated  in  Figures 
4-3  and  4-4  modified  by  the  previous  con¬ 
siderations  for  module  tests  and  the  following: 

a.  Testing  the  total  man-machine  interface 
using  the  actual  system  hardware  and 
support  software  in,  as  nearly  as  practi¬ 
cable,  actual  operating  conditions  to 
provide  Certification. 

b.  Testing  of  the  system  as  prescribed  by 
the  system  specification,  to  include  use 
of  actual  operational  manuals  to  load 
and  initiate  program(s)  and  operate 
the  system;  demonstrate  control  of  the 
system  from  all  local  and  remote  modes 
of  operation,  capability  of  performance 
monitoring,  diagnostic,  degraded  and 
casualty  mode  techniques. 

c.  Testing  of  system  internal  and  external 
software  interfaces  (in  SSPO  termin¬ 
ology,  these  are  all  inte  ll  subsystem 
interfaces  and,  to  the  extent  practicable 
or  allowable  by  contract,  external  inter¬ 
faces  with  other  subsystems)  utilizing 
actual  hardware,  system  software,  and 
operational  manuals;  includes  all  perfor¬ 
mance  monitoring,  failure  detection  and 
isolation,  degraded  and  casualty  mode 
techniques  (reaction  to  missing,  illegal, 
and  “significantly  different  to  previous” 
inputs,  internal  and  external  computer 
communications). 

d.  Testing  of  software  system’s  ability  to 
perform  under  stress  (proper  reactions 


and  response  times)  as  a  result  of  data 
rate  saturation  with  legal  and  illegal 
inputs,  data  transfer  overload,  simul¬ 
taneously  (maximum  use  of  data  input/ 
output  ports)  exercising  all  sections  of 
the  program,  by  overloading  main  and 
secondary  storage  (i.e.,  mass  and  scratch 
memories,  tables,  buffers),  simultaneous 
operation  and  scenario-operation  of  all 
hardware. 

e.  Interconnecting  control  and  data  path 
diagrams  should  appear  in  test  docu¬ 
ments. 

f.  Conduct  of  the  test  should  include: 

(1)  Failure  definition  generally  based  on 
specified  performance,  with  no  stop¬ 
pages  or  interruptions  due  to  soft¬ 
ware  errors.  (Use  of  self-recovery 
features  frequently  reflects  an  under¬ 
lying  software  failure  and  should  be 
treated  as  a  program  stop). 

(2)  Length  of  test  based  upon:  contin¬ 
uous  or  less  than  continuous  operat¬ 
ing  programs,  saturation  testing,  re¬ 
duced  capability  testing  (shutting  off 
equipments),  manual  and  automatic 
operation  testing,  scenario  testing 
(modes  of  operation),  failure  com¬ 
pensation  testing  (including  failures). 

(3)  Recording  of  all  events  (see  NAVSEA 
OD29304B  for  example  of  form), 
failures/errors,  response  capability, 
shutdown,  restart  and  recovery  tech¬ 
niques,  degraded  and  casualty  capa¬ 
bilities,  etc. 

(4)  Corrective  action  plan  of  action  to 
isolate  failure  to  hardware  or  soft¬ 
ware,  steps  to  be  followed  to  deter¬ 
mine,  initiate  and  verify  corrective 
action,  while  avoiding  inducement  of 
other  errors. 

g.  Demonstration  that  the  software  pro- 
gram(s)  can  be  modified  as  may  be 
required  in  service,  i.e.,  use  of  latest 
revisions  of  all  documentation,  pro¬ 
cedures  adequate  to  implement  program 
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modifications  (i.e.,  error  correction  re¬ 
sponding  to  changes  in  requirements 
or  the  operational  environment),  pro¬ 
cedures  in  place  to  verify  modification 
and  that  all  other  impacted  software  has 
been  analyzed  and  corrected. 

4.4  PLANNING  RELIABILITY 
DEMONSTRATION  TESTS 

Reliability  demonstration  testing  has  a 
specific  meaning  in  a  test  program.  It  is  a 
formal  test,  of  significant  duration  and  sample 
size,  to  demonstrate  a  specification  require¬ 
ment  statistically.  Reliability  tests  should  be 
performed  over  the  complete  range  of  design 
environments,  utilizing  extreme  or  worst  case 
operational  stress  levels  in  the  sequence  en¬ 
countered  in  the  mission.  Items  under  test 
should  be  representative  of  production  items, 
manufactured  by  the  same  manufacturing 
techniques  that  will  be  used  for  the  production 
items,  receiving  the  same  bum-in  or  condition¬ 
ing  that  production  items  will  receive,  and 
containing  all  approved  design  modifications. 

The  design  of  an  item  is  based  on  require¬ 
ments  defined  in  one  or  more  specification 
documents.  A  reliability  test  should  provide 
confidence  that  the  item  can  meet  the  re¬ 
liability  requirements,  either  by  testing  for  an 
extended  length  of  time  (or  number  of  cycles) 
or  by  testing  a  statistically  determined  quan¬ 
tity  of  items.  Tear-down  and  visual  inspection 
may  be  needed  after  the  tests  to  verify  the 
item  condition.  This  length  of  time  or  quan¬ 
tity  of  items  is  the  major  difference  between 
reliability  tests  and  general  qualification  tests. 
Design  constraints  generally  encompass  the 
normal  operating  conditions  an  item  is  ex¬ 
pected  to  see.  For  example,  an  item  which  is 
designed  to  operate  over  the  temperature 
range  of  0->0‘’(  ,  but  will  normally  operate 
between  25  and  30°C,  should  be  tested  to 
provide  assurance  that  the  item  can  operate 
over  the  0-50”C  range.  This  will  assure  that 
the  item  can  operate  in  the  normal  tempera¬ 
ture  range,  while  also  assuring  that  the  item 
wiil  operate  when  the  temperature  conditions 
are  outside  the  normal  range  but  within  the 
design  range.  When  items  are  designed  to  meet 
combinations  of  environmental  conditions, 
such  as  vibration,  humidity,  temperature,  etc.. 


the  reliability  test  should  be  performed  under 
these  combinations  of  conditions  The  se¬ 
quence  of  the  environments  and  the  sequence 
of  combinations  of  environments  should 
follow  the  mission  profile. 

When  the  test  is  time  or  cycle  oriented, 
trade-offs  have  to  be  made  as  to  the  number 
of  items  to  be  placed  under  test.  Trade-off 
considerations  include:  contract  requirements, 
military  standard  requirements,  program 
schedule,  items  available,  facilities  to  simul¬ 
taneously  handle  multiple  items,  number  of 
test  hours  or  cycles  required  to  be  accumu¬ 
lated,  reliability  prediction  results,  etc.  The 
number  of  items  tested  should  also  depend  on 
the  duration  of  testing  applied  to  a  specific 
item.  The  longer  an  item  is  on  test,  the  more 
knowledge  is  accumulated  about  the  long 
term  effects  of  environmental  conditions, 
interfaces,  and  operation  of  the  item.  A  larger 
sample,  however,  gives  more  information 
about  the  manufacturing  process  that  pro¬ 
duced  the  item,  and  the  performance  of  the 
item  population. 

When  tests  are  destructive  in  nature,  which 
is  the  case  for  use-expended  (one-shot)  items, 
cost  must  be  considered  in  the  trade-off 
analysis. 

When  considering  applying  operational 
interfaces  to  the  item(s)  under  test,  simula¬ 
tion  and  stimulation  techniques  should  be 
completely  analyzed.  Caution  should  be  exer¬ 
cised  because  these  techniques  tend  to  provide 
“pure”  or  “perfect”  inputs  and  outputs.  As 
an  example,  a  real  functioning  synchro,  pro¬ 
viding  real  signals,  will  contain  transients 
which  the  input  of  a  signal  converter  must  be 
designed  to  handle,  however  an  input  signal 
by  simulation  techniques  would  not  contain 
these  transients.  A  test  without  real  synchro 
signals  may  not  be  a  representative  test. 

Appropriate  measures  must  be  taken  to 
insure  limited  access  to  reliability  test  areas, 
to  detect  tampering  with  items  under  test 
(e.g.,  by  use  of  detection  stickers)  and  to  in¬ 
form  test  personnel  about  requirements  to  be 
adhered  to  in  the  test  area.  Test  equipment 
should  not  be  tampered  with  or  removed 
from  the  area.  Adjustments  and  maintenance 
not  allowed  by  test  procedure  must  be  pre¬ 
vented  and  every  test  event  must  be  com- 
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pletely  recorded.  Understanding  of  the  test 
requirements  is  essential  when  designing  test 
facilities.  To  prevent  environmental  stresses 
from  exceeding  the  design  range,  a  facilities 
safety  design  should  be  incorporated  to  en¬ 
sure  the  shut-down  of  any  malfunctioning 
item.  For  example,  limit  switches  which  can 
shut  off  a  malfunctioning  vibration  table  not 
only  save  manpower,  but  in  conjunction  with 
vibration  chart  recorders,  can  maintain  the 
integrity  of  the  test. 

Use  of  existing  or  temporary  facilities  can 
reduce  costs.  Rather  than  purchasing  an  addi¬ 
tional  environmental  chamber  for  one-time 
use,  scheduling  changes  can  be  made  or  con¬ 
sideration  given  to  building  a  generic  test  set¬ 
up  able  to  meet  requirements.  As  an  example, 
to  provide  a  high  temperature  test,  a  tem¬ 
porary  structure  of  wood,  insulation,  circula¬ 
tion  and  exhaust  fans,  heating  elements  and 
temperature  monitoring  sensors  might  be  used 
in  place  of  a  new  environmental  chamber.  Use 
of  independent  test  facilities  may  also  provide 
a  significant  cost  savings. 

Test  equipment  used  for  inputs,  outputs 
and  test  monitoring  must  meet  accuracy, 
calibration  and  data  requirements.  Whether 
the  equipment  is  mechanical,  hydraulic,  elec¬ 
tronic  or  electromechanical,  the  requirements 
must  be  adhered  to  and  addressed  in  the  test 
procedures.  Where  software  programs  are  util¬ 
ized  for  performance  monitonng,  verification 
is  needed  prior  to  the  start  of  the  tests,  to 
assure  that  the  software  program  will  reliably 
detect  degradations  or  failures. 

Digital  readout  or  automated  recording 
equipment,  as  opposed  to  instruments  requir¬ 
ing  visual  interpretation  of  readings,  are  to  be 
preferred  if  available. 

Reliability  tests  should  be  performed  at  the 
highest  assembly  levels  practicable.  An  advan¬ 
tage  of  higher  level  tests  is  that  they  improve 
the  validity  of  the  tests;  the  equipment  is 
operated  close  to  its  mission  mode.  A  disad¬ 
vantage  is  that  the  tests  must  occur  later  in 
the  program  than  would  lower  level  tests,  and 
some  internal  component  states  may  be  diffi¬ 
cult  to  observe. 

Throughout  an  ITP.  test  data  are  generated 
which  may  be  of  use  in  evaluating  reliability. 
When  Figure  4-1  is  completed,  the  ITPB 


should  evaluate  the  proposed  test  for  use  as 
reliability  data.  If  the  test  data  generated  are 
not  usable  for  reliability  demonstration,  it 
may  be  possible  to  modify  the  test  (see 
Sheet  1  of  Figure  4-1)  so  that  the  data  will  be 
suitable.  To  be  considered  for  use  as  reliabil¬ 
ity  data,  the  test  must  be  controlled  per 
approved  procedures,  be  witnessed  by  the 
responsible  organizations,  use  the  proper  test 
items  and  meet  the  interface  and  environ¬ 
mental  requirements. 

Figure  4-1  should  be  completed  by  the  line 
organization  and  submitted  to  the  ITPB  and 
the  facilities  manager,  providing  all  the  details 
of  the  proposed  reliability  test.  Any  differ¬ 
ences  or  variations  between  the  details  on  this 
figure  and  the  specification  requirements 
should  be  fully  identified  and  addressed 
during  the  ITPB  review  meetings,  and  any 
remaining  variations  evaluated  for  expected 
effects  on  the  item’s  demonstrated  reliability. 
NAVSEA  OD29304B  provides  technical  guid¬ 
ance  for  planning  reliability  demonstration 
tests. 

Reliability  test  plans  and  procedures  are 
necessary  to  govern  the  performance  of  the 
tests.  The  test  plan  should  specify  the  statis¬ 
tical  test  plan  chosen,  values  for  risk,  lower 
and  upper  test  parameters  (e.g.,  MTBF),  cite 
applicable  military  specifications  and  satisfy 
the  content  requirements  of  Figure  4-3.  Test 
procedures  should  satisfy  the  content  require¬ 
ments  of  Figure  4-4. 

Reliability  test  reports  should  be  prepared 
reporting  the  results  of  the  tests.  A  test  report 
should  satisfy  the  content  requirements  of 
Figure  4-5.  For  lengthy  tests,  periodic  interim 
reports  should  be  submitted  to  the  ITPB. 
Periodic  summary  reports  should  satisfy  the 
content  requirements  of  Figure  4-6. 

4.5  PLANNING  MAINTAINABILITY 
DEMONSTRATION  TESTS 

Maintainability  demonstration  testing  is 
formal  testing  to  verify  achievement  of  con¬ 
tractual  maintainability  requirements.  Main¬ 
tainability  tests  should  be  performed  on  items 
representative  of  the  production  item  (i.e., 
containing  all  approved  design  modifications), 
with  proven  performance  monitoring  and 
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diagnostic  software  programs,  with  verified 
and  validated  item  documentation  (i.e.,  oper¬ 
ation  manual,  technical  manual),  with  actual 
tools  and  test  equipment  to  be  used  in  service, 
by  personnel  trained  in  the  service  training 
program  and  in  environments  (i.e.,  tempera¬ 
ture,  spatial  constraints)  simulating  those 
which  will  be  found  aboard  the  submarine. 
MIL-STD-471A  provides  technical  guidance 
for  planning  maintainability  demonstration 
tests.  The  following  discusses  some  of  the 
more  general  aspects  of  maintainability  testing. 

The  results  of  a  maintainability  test  should 
provide  statistical  confidence  in  the  maintain¬ 
ability  design  of  the  item  and  a  qualitative 
assessment  of  the  item  maintainability  ap¬ 
proach.  Figure  4-7  is  an  example  of  a  log  for 
recording  qualitative  and  quantitative  data 
during  a  maintainability  test. 

The  maintainability  design  criteria  of  an 
item  is  derived  from  quantitative  require¬ 
ments  and  design  practices.  The  requirements/ 
design  practices  are  based  on: 

a.  Types  of  repairs  and  replacements 
allowed  (and  sparing  concept)  at  each  mainte¬ 
nance  level. 

b.  Manpower  skills. 

c.  Quantity  of  manpower  that  will  be  avail¬ 
able  to  maintain  the  item. 

d.  Maintenance  time  limitations. 

e.  Restrictions  on  special  tools  and  support 
equipment. 

f.  Soldering/unsoldering  limitations. 

g.  Testability  provisions. 

h.  Test  point  provisions. 

i.  Spares  provisions  -  transportation,  pack¬ 
aging,  handling,  preservation. 

j.  Accessibility  provisions. 

k.  Safety  -  handling  constraints  (weight, 
etc.),  hazard  conditions/warnings,  finished 
product  maturity  (sharp  edges,  etc.). 

l.  Preventive /planned /scheduled  mainte¬ 
nance  constraints. 

m.  Mounting/attachment  provisions. 

n.  Built-in-test  provisions  and  require¬ 
ments  -  modularity,  automation,  failure  de¬ 
tection  and  isolation  constraints,  functional¬ 
ity,  testability,  fail  safe  constraints,  diagnostic 
programs,  performance  monitoring  programs. 

o.  Adjustment/ Alignment/Calibration  pro¬ 
visions. 


A  checklist  should  be  utilized  to  assess  all 
of  the  maintainability  characteristics  as  each 
maintenance  task  is  performed.  Tlu  analysis 
should  also  evaluate  the  adequacy  of  person¬ 
nel  training,  supporting  documentation,  tools, 
maintenance  aids,  support  equipment  and 
manufacturing  techniques. 

Test  maintenance  personnel  should  be 
Navy  technicians,  having  the  proper  rating  or 
Navy  Enlisted  Classification,  and  of  the  low¬ 
est  rate  authorized  to  perform  the  item  main¬ 
tenance.  The  technicians  should  he  trained  in 
the  approved  training  program  and,  prefer¬ 
ably,  have  completed  it  several  weeks  before 
the  test.  This  places  the  burden  of  the  mainte¬ 
nance  action  on  the  maintenance  documents 
and  aides,  as  it  should.  If  Navy  technicians 
are  not  available,  contractor  personnel  may 
perform  the  tests  with  the  following  precau¬ 
tions:  1)  these  personnel  should  not  be  at  an 
advantage  or  disadvantage  when  performing 
the  test,  2)  the  test  team  should  verify  that 
the  technicians  follow  the  maintenance 
manual  procedures  step  by  step. 

Documentation  and  maintenance  aides 
must  be  in  versions  that  have  been  verified 
and  validated.  When  developing  the  docu¬ 
mentation,  certain  basic  premises  must  be 
met,  i.e.,  material  must  be  sufficiently  de¬ 
tailed,  must  be  written  for  the  proper  level  of 
technician  training,  and  must  have  been  writ¬ 
ten  by  personnel  knowledgeable  in  the  oper¬ 
ation  and  maintenance  of  the  item.  Support 
equipment  should  be  that  equipment  called 
for  in  the  maintenance  manuals.  Support 
equipment  should  be  assessed  for  capability, 
proper  maintenance  and  calibration.  Tools 
should  include  only  those  called  for  in  the 
maintenance  manuals.  Spares  should  be  sup¬ 
plied  to  the  maintenance  technician  as  they 
would  be  supplied  on  the  submarine. 

The  item  under  test  should  be  as  similar 
to  the  production  item  as  possible.  AH  known 
modifications  and  corrective  actions  should 
be  incorporated  into  the  test  item.  Software 
programs  should  be  patch  free,  verified  and 
validated. 

The  1TPB  should  assure  a  pretest  evaluation 
of  the  complete  maintenance  package.  Areas 
of  concern  are:  When  simulating  or  inducing 
faults  (such  as  prefaulted  modules)  into  the 
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PERIODIC  TEST  SUMMARY  REPORT 

The  Periodic  Test  Summary  Report  should,  as  a  minimum,  include  the  following: 

a.  Type  and  number  of  items  on  test  and  type  of  test. 

b.  Total  elapsed  item  hours  (cycles),  test  hours  (cycles)  during  the  period 
covered  in  the  report  and  the  total  accrued  item  hours  (cycles)  of  testing  at 
the  report  date. 

c.  Total  number  of  item  failures  for  each  operational  mode  specified  in  the 
duty  cycle. 

d.  Description  of  each  failure  problem  area,  related  failure  analysis,  and 
corrective  action. 

e.  Test  conditions  and  analysis  of  any  variation  from  specified  conditions. 

f.  Present  reject/accept  status  and  estimated  test  time  (cycles)  for  completion 
and  date. 

g.  A  chart  showing  a  plot  of  the  observed  reliability  characteristic  from  start 
of  test  through  the  report  period  and  the  predicted  value  for  comparison. 

h.  A  detailed  chart  depicting  the  history  of  every  item  under  test  by  serial 
number  and  with  time  (meter  readings)  or  cycles  recorded.  This  history 
traces  all  activities  from  and  including  pre-conditioning  to  delivery. 

i.  All  pre-conditioning  time  and  failures  by  item  serial  number  on  all  items 
delivered. 

j.  Cumulative  results  of  previous  and  present  month's  testing  under  present 
contract. 

k.  The  status  and/or  disposition  of  each  corrective  action. 


Figure  4-6.  Typical  Content  Of  A  Periodic  Test  Summary  Report 


test  item,  the  technique  should  actually  be 
checked  out  for  “proper”  operation  of  the 
fault,  and  the  maintenance  documentation 
should  be  verified  against  the  test  item.  Since 
the  total  document  package  should  have  been 
verified  previously,  the  impact  of  an  error 
found  in  this  portion  of  pretest  should  be 
carefully  considered,  leading  to  a  complete 
review  of  the  document.  The  test  program 
should  make  allowances  for  time  to  perfect 
and  verify  corrective  actions  of  problems 
found  during  the  pretest  evaluation.  The  per¬ 
formance  of  the  pretest  evaluation,  and  the 
verification  of  corrective  action  implementa¬ 
tion,  should  assure  that  when  maintainability 
tests  are  performed,  valid  test  data  are  ob¬ 
tained,  within  reasonable  time  constraints  and 
without  the  constant  test  interruptions  so 


often  associated  with  maintainability  testing. 
Some  other  areas  to  be  considered  are: 

a.  What  happens  if  a  test  does  not  include 
enough  events  to  be  statistically  valid?  As  an 
example,  as  items  are  designed  to  require  less 
preventive  maintenance  tasks,  it  is  not  always 
practical  or  reasonable  to  test  statistically  the 
design  of  these  tastes.  However,  during  a  test 
the  design  of  and  adequacy  of  the  mainte¬ 
nance  package  for  these  tasks  should  be 
assessed.  Generally,  what  should  be  done  is  to 
perform  the  maintenance  on  the  taskist 
enough  times  to  assure  the  test  team  that  the 
design  is  satisfactory  and  that  all  documenta¬ 
tion.  training,  etc  is  adequate  When  repeat 
ine  the  same  task,  allowances  must  be  made 
for  the  learning  curse  ot  the  m.imten.uue 
teehmeia  n 
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b.  As  in  all  formal  testing,  the  integrity 
of  the  test  must  be  maintained,  procedures 
strictly  adhered  to,  the  area  restricted  to  the 
test  team,  and  distractions  and  noise  beyond 
normal  levels  kept  to  a  minimum. 

c.  Qualitative,  and  to  some  extent  quanti¬ 
tative,  deficiencies  discovered  during  the  test 
should  be  corrected  as  they  are  found.  This  is 
not  always  possible,  but  every  effort  should 
be  made  to  make  and  verify  the  corrections. 
If  the  deficiencies  become  too  numerous, 
then  it  is  the  obligation  of  the  test  team  to 
terminate  the  test  and  perform  corrective 
actions.  The  complete  maintenance  package 
should  again  be  verified  prior  to  retesting  or 
restarting  the  test. 

d.  Environmental  constraints  and  condi¬ 
tions  during  the  maintainability  test  should  be 
consistent  with  operational  conditions.  Areas 
to  be  considered  include  confinement  of 
maneuvering  space,  accessibility  to  the  item, 
and  orientation  of  the  item,  as  well  as  temper¬ 
ature,  vibration,  pressure  and  noise  environ¬ 
ments  that  would  present  problems  to  the 
normal  work  effort  or  require  extra  or  special 
clothing. 

Planning  for  a  maintainability  test  begins 
with  the  submittal  of  test  details  to  the  ITPB 
by  the  line  organization,  using  Figure  4-1. 
Any  differences  or  variations  between  the 
details  on  this  figure  and  the  specification 
requirements  should  be  fully  identified  and 
addressed  during  the  ITPB  review  meetings; 
any  remaining  variations  should  be  evaluated 
for  expected  impact  on  the  item’s  demon¬ 
strated  maintainability. 

Maintainability  test  planning  must  consider 
not  only  the  demonstration  of  a  maintain¬ 
ability  requirement,  such  as  mean  time  to 
repair  (MTTR),  but  also  the  verification  of 
the  appropriate  testability  requirements, 
which  have  an  impact  on  operational  avail¬ 
ability  and  maintainability.  The  testability 
parameters  which  are  usually  specified  include 
fault  coverage,  false  alarm  rate,  and  fault 
isolation.  The  requirements  for  fault  coverage 
and  false  alarm  rate  are  often  written  in  a 
form  such  as:  “The  system  shall  detect  98% 
of  operational  faults  with  a  false  alarm  rate 
not  to  exceed  2%.  The  system  shall  isolate  a 
fault  to  a  single  module  90%  of  the  time,  and 


to  an  ambiguity  group  of  not  more  than  5 
modules  99%  of  the  time.” 

The  first  two  parameters,  fault  coverage 
and  false  alarm  rate,  are  usually  verified  by 
analysis  of  the  circuitry,  including  Failure 
Mode.  Effects  and  Criticality  Analysis 
(FMECA).  NAVSEA  OD29304B  provides 
technical  guidance  for  FMECA.  If  false  alarm 
rate  is  specified  in  time  e.g.,  mean  time  be¬ 
tween  fahe  alarms,  a  MIL-STD-78!  procedure 
(where  a  false  alarm  is  considered  a  failure) 
can  he  employed  to  demonstrate  the  require¬ 
ment. 

Verification  of  the  fault  isolation  require¬ 
ment  is  a  joint  analysis  and  test  function.  The 
testability  or  maintainability  analysis  assures 
that  the  hardwarc/software  design  allows  for 
the  proper  number  and  placement  of  test 
points  to  isolate  the  detected  fault.  This 
analysis  is  a  major  input  to  the  critical  design 
review  and  should  be  used  when  preparing 
software  element  and  integration  test  specifi¬ 
cations. 

The  actual  demonstration  of  the  isolation 
capability  of  the  design  with  respect  to  the 
requirements  occurs  during  the  maintain¬ 
ability  demonstration.  Here  the  selection  of 
faults  plays  an  important  role  in  determining 
if  the  test  will  exercise  the  diagnostic  software 
adequately.  The  faults  must  be  selected  so 
that  modules  which  will  fail  most  often  are 
faulted  most  often.  However,  it  is  necessary 
to  avoid  faulting  a  module  in  an  identical 
manner  if  the  module  is  selected  again.  This 
modification  to  the  selection  process  allows 
wider  exercise  of  diagnostic  programs  and 
maintenance  manuals. 

Maintainability  test  plans  and  procedures 
are  needed  to  govern  the  performance  of  the 
test.  Test  plans  and  procedures  should  satisfy 
the  content  requirements  of  Figures  4-8  and 
4-9.  respectively.  Maintainability  test  reports 
should  satisfy  the  content  requirements  of 
Figure  4-10. 

4  6  WEAPON  SYSTEM  TESTS 

4.6.1  Weapon  System  Integrated 
Test  Program 

A  Weapon  System  Integrated  Test  Program 
is  performed  under  SSPO  direction.  Tests  cov- 
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MAINTAINABILITY  TEST  PLAN 

The  Maintainability  Test  Plan  should  identify  and  describe  planned  contractor  activities 
for  implementation  of  test  requirements.  The  plan  should,  as  a  minimum,  include  the 
following,  as  appropriate: 

a.  The  content  of  Figure  4-3. 

b.  A  description  of  the  maintenance  concept. 

c.  Identification  of  the  level(s)  of  maintenance  to  be  demonstrated. 

d.  A  description  of  the  adequacy/inadequacies  of  maintenance  support  ele¬ 
ments  (documentation,  tools,  spares,  etc.). 

e.  The  qualifications  and  training  (experience  resumes)  of  operator  and 
maintenance  personnel  performing  the  tests. 

f.  Discussion  of  test  team  personnel  indoctrination. 

g.  The  procedure  for  selection  of  maintenance  tasks. 

h.  The  identification  of  "Special''  maintenance  tasks. 

Figure  4-8.  Typical  Content  Of  A  Maintainability  Test  Plan 


MAINTAINABILITY  TEST  PROCEDURE 

The  Maintainability  Test  Procedure  should  provide  technical  directions  for  implement¬ 
ing  the  required  test.  Procedures  contain  step-by-step  instructions  of  how  the  test  will 
be  set  up,  initiated,  performed,  and  secured.  The  test  procedure  should  as  a  minimum, 
include  the  following,  as  appropriate: 

a.  The  content  of  Figure  4-4. 

b.  The  method  of  handling  outstanding  pretest  problems. 

c.  The  number  of  hours  the  maintenance  technician(s)  will  work  each  day. 

d.  The  method  of  handling  test  interruption  (i.e.,  deficient  documentation, 
training,  software  programs,  etc.). 


Figure  4-9.  Typical  Content  Of  A  Maintainability  Test  Procedure 
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MAINTAINABILITY  TEST  REPORT 

The  Maintainability  Test  Report  is  the  formal  record  of  the  results  of  an  item's  main¬ 
tainability  test.  The  test  report  should,  as  a  minimum,  include  the  following,  as  appro¬ 
priate: 

a.  The  content  of  Figure  4-5. 

b.  The  results  of  the  pretest  evaluation. 

c.  An  analysis  of  qualitative  observations  made  during  the  test  (such  as  a 
matrix  of  deficiencies  against  item  design,  item  manufacture,  documenta¬ 
tion,  training,  software  programs,  etc.). 

d.  The  intended  action  and  time  frame  to  rectify  all  deficiencies  (i.e.,  "will  be 
redesigned",  "will  be  redesigned  during  production",  "pen  and  ink  changes 
will  be  proposed",  etc.). 


Figure  4-10.  Typical  Content  Of  A  Maintainability  Test  Report 


ered  by  the  Weapon  System  Integrated  Test 
Program  include  missile  flight  tests  (which 
actually  begin  in  the  development  phase), 
DASO  tests,  patrol  tests,  OT/FOT  tests  and 
surveillance  tests,  including  those  of  the  Fleet 
Return  Evaluation  Program. 

The  general  purpose  of  the  system  level  test 
program  is  to: 

1)  Determine  the  weapon  system’s  readi¬ 
ness  for  operational  deployment. 

2)  Verify  processes  within  the  weapon  sys¬ 
tem  that  determine  its  performance  capabili¬ 
ties  and  to  enable  valid  extrapolation  of 
performance  observed  under  test  conditions 
to  performance  expected  under  operational 
conditions. 

3)  Provide  for  the  Commander-in-chiefs 
estimates  of  performance  planning  factors 
under  operational  patrol  conditions,  and  to 
make  this  information  available  to  the  JCS 
at  the  earliest  possible  date  for  use  in  SIOP 
planning. 

4)  Provide  continuing  evaluations  of 
sources  of  inaccuracy,  unreliability  and 
maintenance  burden  within  the  weapon 
system,  and  to  guide  improvement  efforts. 

Documents  developed  for  the  test  program 
include: 

1)  A  Weapon  System  Integrated  Test  Pro¬ 
gram  Plan. 

2)  A  Weapon  System  Analysis  Plan. 

3)  A  DASO  Test  Plan. 


4)  A  DASO  Analysis  Plan. 

5)  DASO  Test  Reports. 

6)  A  OT  Test  Plan. 

7)  A  OT  Analysis  Plan. 

8)  OT/FOT  Performance  Reports. 

9)  A  Patrol  Test  Plan. 

10)  A  Patrol  Analysis  Plan. 

1 1)  Patrol  Performance  Reports. 

1 2)  Flight  Trajectory  Reports. 

The  Weapon  System  Integrated  Test  Pro¬ 
gram  Plan  describes  the  planned  weapon  sys¬ 
tem  tests  and  indicates  how  test  results  will 
be  used.  The  Weapon  System  Integrated  Test 
Program  Plan  describes  all  system  evaluations 
to  be  performed  as  a  result  of  the  planned 
tests,  addresses  range  safety  and  takes  into 
consideration  requirements  for  test  facilities, 
including  any  new  facilities  that  may  be 
needed  to  support  the  tests. 

The  measures  of  deployed  system  perfor¬ 
mance  evaluated  during  these  tests  are  reliabil¬ 
ity,  accuracy,  communications  and  reaction 
time,  firing  rate  and  the  envelope  of  perfor¬ 
mance  (flight  range  and  loft  and  footprint 
capability).  Test  data  acquired  are  analyzed  to 
determine  estimates  of  each  of  the  constituent 
elements  of  weapon  system  reliability  and 
contributors  to  inaccuracy  during  the  test. 
Errors  are  grouped  by  type  and  combined 
using  a  mathematical  system  error  model  to 
propagate  them  through  the  system  to  reentry 
body  impact.  Sensitivity  coefficients  are  used 
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to  calculate  the  miss-distance  contributions  of 
subsystems  and  components.  Three  types  of 
errors  predominate: 

1)  Measurement  (Input)  Errors  -  Errors  in 
determining  or  stating  quantities  used  in 
generating  control  functions. 

2)  Scheme  (Formulation  Model)  Errors  ~ 
Errors  resulting  from  use  of  imperfect  rela¬ 
tionships  or  equations  in  generating  control 
functions. 

3)  Variation  (Output)  Errors  -  Errors  that 
occur  when  control  functions  generated  by 
the  system  are  not  identical  each  time  they 
are  replicated  using  the  same  input  func¬ 
tions. 

Subsystem  Accuracy  is  evaluated  as  fol¬ 
lows: 

1)  Navigation  subsystem  accuracy  is  eval¬ 
uated  in  terms  of  discrete  launch  errors,  indi¬ 
vidual  equipment  accuracy  statistics  and  long¬ 
term  RMS  errors.  Navigation  errors  include 
latitude  and  longitude  errors,  heading  and 
velocity  errors,  and  time  of  day  errors. 

2)  Fire  control  subsystem  accuracy  is 
measured  in  terms  of  errors  in  guidance  plat¬ 
form  position,  velocity,  orientation  at  launch, 
time  of  flight,  stored  coefficients,  guidance 
system  alignment  within  the  tube  and  with 
respect  to  optical  navigation  reference. 

3)  Missile  subsystem  errors  relate  to  guid¬ 
ance  components,  separation  velocity  and 
gas  dynamics  variations,  drag  coefficients, 
weather  parameters  during  reentry  and  errors 
of  arming  and  fuzing  components. 

4.6.2  Planning  for  Weapon  System 
Integrated  Test  Program 

The  subsystem  contractor’s  major  respon¬ 
sibility  for  contributing  to  the  Weapon  System 


Integrated  Test  Program  is  to  participate  • 
SSPO’s  Data  Requirements  Working  Group 
and  supply  information  documentation  giving 
subsystem  description  and  functional  perfor¬ 
mance.  to  the  extent  required  by  contract. 
Numerous  inputs  to  DASO,  patrol  and 
OT/FOT  planning  tasks  may  originate  with 
subsystem  contractors.  These  inputs  include 
subsystem  accuracy  and  reliability  models, 
the  specific  function  measurements  required 
to  evaluate  subsystem  performance,  as  well 
as  the  degree  of  measurement  accuracy,  pre¬ 
cision  and  repetition  rate  required.  Data  ele¬ 
ments  the  subsystem  contractor  should  plan 
to  provide  include:  each  function  to  be  mea¬ 
sured,  tests  in  which  the  function  is  to  be 
measured,  source/pickoff,  up-date  rate,  accu¬ 
racy  (/i,  o,  RMS),  precision/resolution,  band¬ 
width,  range,  sample  rate,  maximum  rate  of 
change,  data  time  correlation,  and  time  interval. 

Also  to  the  extent  required  by  the  con¬ 
tract,  the  subsystem  contractor  will  perform 
evaluation  of  data  from  the  weapon  system 
tests. 

Many  decisions  on  inputs  to  the  Weapon 
System  Integrated  Test  Plan  and  evaluation 
of  weapon  system  test  results  are  aided  by 
data  from  the  development  and  production 
test  program.  Planning  includes: 

1)  Assuring  that  all  essential  performance 
parameters  that  have  been  definitized  are 
verified. 

2)  Assuring  that  models  are  defined  in 
terms  of  observables. 

3)  Assuring  that  enough  development  and 
production  test  data  is  available  to  evaluate 
the  results  of  weapon  system  tests. 
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